Automated detection of null field values and effectively null field values

ABSTRACT

Disclosed are systems for, and methods of, automatically detecting and treating field values of a particular field as null field values in records of a database. The system and method provide automatic treatment of these field values as null field values by calculating a critical frequency for the field. Based on the critical frequency of the field, the system and method treats field values that occur more than the critical frequency of the field as null field values and treats field values that occur less than the critical frequency as non-null field values.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims priority to and hereby incorporates byreference in their entireties U.S. Provisional Patent Application No.61/047,570 entitled “Database systems and methods” to Bayliss filed Apr.25, 2008 and U.S. Provisional Patent Application No. 61/053,202 entitled“Database systems and methods” to Bayliss filed May 14, 2008. Theseapplications are referred to herein as the “Second Generation PatentsAnd Applications.”

The following patents and patent applications are related to the presentdisclosure and are hereby incorporated by reference in their entireties:

-   -   U.S. Pat. No. 7,293,024 entitled “Method for sorting and        distributing data among a plurality of nodes” to Bayliss et al.;    -   U.S. Pat. No. 7,240,059 entitled “System and method for        configuring a parallel-processing database system” to Bayliss et        al.;    -   U.S. Pat. No. 7,185,003 entitled “Query scheduling in a        parallel-processing database system” to Bayliss et al.;    -   U.S. Pat. No. 6,968,335 entitled “Method and system for parallel        processing of database queries” to Bayliss et al.;    -   U.S. patent application Ser. No. 10/357,447 entitled “Method and        system for processing data records” to Bayliss et al.;    -   U.S. patent application Ser. No. 10/357,481 entitled “Method and        system for linking and delinking data records” to Bayliss et        al.;    -   U.S. patent application Ser. No. 10/293,482 entitled        “Global-results processing matrix for processing queries” to        Bayliss et al.;    -   U.S. patent application Ser. No. 10/293,475 entitled “Failure        recovery in a parallel-processing database system” to Bayliss et        al.;    -   U.S. patent application Ser. No. 10/357,418 entitled “Method and        system for processing and linking data records” to Bayliss et        al.;    -   U.S. patent application Ser. No. 10/357,405 entitled “Method and        system for processing and linking data records” to Bayliss et        al.;    -   U.S. patent application Ser. No. 10/357,489 entitled “Method and        system for associating entities and data records” to Bayliss et        al.;    -   U.S. patent application Ser. No. 10/357,484 entitled “Method and        system for processing data records” to Bayliss et al.;    -   U.S. patent application Ser. No. 11/671,090 entitled “Query        scheduling in a parallel-processing database system” to Bayliss        et al.;    -   U.S. patent application Ser. No. 11/772,634 entitled “System and        method for configuring a parallel-processing database system” to        Bayliss et al.; and    -   U.S. patent application Ser. No. 11/812,323 entitled        “Multi-entity ontology weighting systems and methods” to        Bayliss.

The above applications are referred to herein as the “First GenerationPatents And Applications.” This disclosure may refer to variousparticular features (e.g., figures, tables, terms, etc.) in the FirstGeneration Patents And Applications. In the case of any ambiguity ofwhat is being referred to, the features as described in U.S. patentapplication Ser. No. 11/772,634 entitled “System and method forconfiguring a parallel-processing database system” to Bayliss et al.shall govern.

FIELD OF THE INVENTION

The invention relates to database systems and methods. Moreparticularly, the invention relates to a technique that provides anumerical critical frequency for a field that may be used to detectfield values that may be treated as null.

SUMMARY OF THE CLAIMED INVENTION

According to various exemplary embodiments of the invention, a systemfor, and method of, detecting and treating one or more field values asnull field values in records of an electronic database, is presented.The technique includes selecting a field of the records. The techniquealso includes determining, using a programmed computer, a frequency foreach field value of the field, the frequency comprising an amount of therecords in which the field value of the field appears. The techniquefurther includes calculating, using a programmed computer, a criticalfrequency for the field based on the frequency of each field value ofthe field. The technique further includes treating, using a programmedcomputer, each field value of the field with a frequency that is greaterthan the critical frequency of the field as a null field value and eachfield value of the field with a frequency that is less than the criticalfrequency of the field as a non-null field value.

Various optional features of the above-described embodiments include thefollowing. The calculating may include ordering the frequencies of thefield values of the field according to frequency along a horizontal axisof a histogram of the one or more field values of the field. Thecalculating may also include calculating the difference between adjacentfrequencies in the histogram. The calculating the may includecalculating f(x)=g(x)−g(x+1), wherein f is a function of the frequencydifferences based on a function g that is defined by the frequency ofeach field value of the field. The critical frequency of the field maybe determined by determining the first point at which a derivative ofthe f function changes from a first sign to a second sign. The criticalfrequency of the field also may be determined by determining the firstpoint at which a derivative of the f function changes from a positivesign to a negative sign. The critical frequency of the field may befurther determined by determining the first point at which a derivativeof the f function changes from a negative sign to a positive sign. Thecritical frequency of the field even further may be determined bydetermining the first point at which a derivative of the f functionequals zero. The critical frequency of the field still may be determinedby determining the first point at which a derivative of the f functiondecreases below a threshold. The treating may include treating eachfield value of the field with a frequency that is equal to the criticalfrequency of the field as a null field value. The technique may includereplacing one or more field values treated as null field values with acanonical null field value.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention, both as to its structure and operation together with theadditional objects and advantages thereof are best understood throughthe following description of exemplary embodiments of the presentinvention when read in conjunction with the accompanying drawings.

FIG. 1 is a flowchart depicting an embodiment of an invention of SectionI.

FIG. 2 is a flowchart depicting an embodiment of an invention of SectionII.

FIG. 3 is a flowchart depicting an embodiment of an invention of SectionIII.

FIG. 4 is a flowchart depicting an embodiment of an invention of SectionIV.

FIG. 5 is a flowchart depicting an embodiment of an invention of SectionV.

FIG. 6A is a flowchart depicting an embodiment of an invention ofSection VI.

FIG. 6B is an exemplary histogram an embodiment of an invention ofSection VI.

FIG. 6C is an exemplary graph according to an embodiment of an inventionof Section VI.

FIG. 6D is an exemplary graph according to an embodiment of an inventionof Section VI.

FIG. 7 is a flowchart depicting an embodiment of an invention of SectionVII.

FIG. 8A is a flowchart depicting an embodiment of an invention ofSection VIII.

FIG. 8B depicts an exemplary portion of a search tree according to anembodiment of an invention of Section VIII.

FIG. 9 is a flowchart depicting an embodiment of an invention of SectionIX.

FIG. 10 is a flowchart depicting an embodiment of an invention ofSection X.

DETAILED DESCRIPTION

The following detailed description presents several inventive concepts,which are inter-related.

The following Table of Contents summarizes the present disclosure.

Table of Contents Section Techniques For Linking Records And EntityRepresentations I Statistical Record Linkage Calibration At The FieldAnd Field Value Levels II Without The Need For Human InteractionStatistical Record Linkage Calibration For Reflexive And SymmetricDistance III Measures At The Field And Field Value Levels Without TheNeed For Human Interaction Statistical Record Linkage Calibration ForReflexive, Symmetric And Transitive IV Distance Measures At The FieldAnd Field Value Levels Without The Need For Human InteractionStatistical Record Linkage Calibration For Interdependent Fields WithoutThe V Need For Human Interaction Automated Detection Of Null FieldValues And Effectively Null Field Values VI Adaptive Clustering OfRecords And Entity Representations VII Automated Selection Of GenericBlocking Criteria VIII Automated Calibration Of Negative Field WeightingWithout The Need For IX Human Interaction Statistical Record LinkageCalibration For Multi Token Fields Without The Need X For HumanInteraction An Exemplary Embodiment XI Conclusion XII

Certain terms used herein are discussed presently. The term “entityrepresentation” encompasses at least one record, and, more typically, acollection of linked records that refer to the same individual. Thisterm is meant to embrace the computer implemented entities of the FirstGeneration Patents And Applications. The term “field” encompasses anyportion of a record into which a field value may be entered. The term“field value” encompasses means and manners used to representinformation, not limited to numerical values. A “field value” mayinclude other types of data values comprising one or more charactertypes or combination of character types. This term is meant to embracethe “data field values” of the First Generation Patents AndApplications. The term “token” encompasses any part of a field value,including the entirety of a field value. The term “individual”encompasses a natural person, a company, a body of work, and anyinstitution. The term “probability” encompasses any quantitative measureof likelihood or possibility, not limited to numerical quantitiesbetween zero and one. The term “record” encompasses any data structurehaving at least one field. This term is meant to embrace the “entityreferences” of the First Generation Patents And Applications. Thediscussion in this paragraph is meant to provide instances of what isembraced by certain terms by way of non-limiting example and should notbe construed as restricting the meaning of such terms.

The present document includes disclosures of several inventions, whichare presented in the following Sections I-XII. Embodiments of theseinventions may interact and work together with each other and with thesystems and methods presented in the First Generation Patents AndApplications. For example, parameters generated by an embodiment of aninvention presented in one section may be used by an embodimentpresented in another section or in the First Generation Patents AndApplications. Exemplary details of such interaction are presentedherein.

I. Techniques for Linking Records and Entity Representations

Embodiments of the techniques presented in this section may be used in adatabase to link records and entity representations. More particularly,this section includes disclosure of techniques that may be used tocompare records and decide whether such records refer to the sameindividual and should be linked. The techniques presented in thissection may be used and integrated with techniques of other sections.

FIG. 1 is a flowchart depicting an exemplary embodiment of an inventionof Section I. In general, embodiments presented in this section mayoperate by comparing field values in common fields of two records. Thecomparisons may be performed in the context of a matching formula (e.g.,Equations 2-5 below). Such comparisons may yield, for each field, aprobability that the field values match. In some embodiments, a givenprobability may be one (1) if the fields exactly match and zero (0)otherwise. Other techniques for generating such probabilities aredisclosed in the First Generation Patents And Applications, e.g., in thecontext of EQs 1 and 2. In general, embodiments of the inventionsdisclosed in this section may calculate, for two records, a weighted sumof such probabilities. That is, each such probability may be multipliedby a weight, and those products of probabilities and weights may then besummed. Certain embodiments of inventions disclosed in this document(e.g., in Sections II, III, IV, V and X) generate weights used in suchweighted sums. That is, certain embodiments presented in this sectionmay utilize weights generated by embodiments presented in othersections. If a weighted sum exceeds a threshold, the compared recordsmay be linked.

Embodiments presented in this section may calculate matching formulasthat utilize weighted sums that take into account existing fields (andfield values) in two records under comparison. However, such embodimentsare not limited to consideration of existing fields (and field values).Certain embodiments presented in this disclosure create new fields (andfield values) that may be used in addition to or instead of existingfields (and field values). That is, the weighted sums presented in thissection may range over existing record fields (or field values), newlyadded record fields (or field values), or a combination of both.

At block 105, the exemplary embodiment calculates match probabilities.The weights generated according to certain embodiments and utilized inthe matching formula weighted sums presented in this section may bederived from certain probabilities, referred to herein as “field valueprobabilities,” “field probabilities” and, collectively, as “matchprobabilities.” For convenience, and throughout this disclosure, aprobability associated with an individual field value will be referredto as a “field value probability.” A probability associated with a fieldrather than a particular field value will be referred to herein as a“field probability.” Both terms will be referred to collectively as“match probabilities.” Exemplary embodiments may produce field valueprobabilities associated with every non-null field value in everyrecord, as well as field probabilities associated with every fieldappearing in any record. Each field value probability may represent theprobability that a record (or entity representation) chosen at randomcontains (respectively, contains a record that contains) the associatedfield value. Each field probability may represent the probability thattwo randomly chosen records (respectively, entity representations) sharea common field value in the associated field (respectively, in theassociated field in included records). In certain embodiments, the matchprobabilities may be produced using an iterative process. An exemplary,non-limiting process is discussed in Section II; note, however, thatsuch process may be combined with other processes presented herein.

At block 110, the exemplary embodiment calculates match weights. Incertain embodiments, the weights utilized in the matching formulaweighted sums presented in this section may be derived from matchprobabilities. The field value probabilities may be converted to fieldvalue weights, and the field probabilities may be converted to fieldweights. As discussed in this section, these weights may be used inweighted sums in order to determine whether to link two records. Aseparate field value weight may be associated with each field valueappearing in any record in the database; however, in some embodimentssuch field value weights may be associated with only a subset of thetotality of field values appearing in any record in the database. Aseparate field weight may be associated with each field appearing in anyrecord in the database; however, in some embodiments such field weightsmay be associated with only a subset of the totality of fields appearingin any record in the database. The terms “field value weights” and“field weights” are referred to collectively herein as “match weights.”In certain embodiments that utilize an iterative process to generatematch probabilities, which may be converted into match weights, eachiteration of such process may produce increasingly accurate matchprobabilities and match weights.

Note that match probabilities, which may be used to derive matchweights, should not be confused with the probabilities that may beweighted by the match weights. That is, the probabilities used to derivematch weights generally referred to herein as w_(i) should not beconfused with the probabilities p_(i), which appear in the matchingformulas presented herein (and in EQs 1 and 2 of the First GenerationPatents And Applications).

Deriving match weights from match probabilities may proceed as follows.Note that the match weights so produced may have the advantage ofallowing for easier computer implementation. Certain computers andprogramming languages may be ill-adapted to handle small numbers (e.g.,products of probabilities lying in the interval (0,1)), without the riskof introduced rounding error. Conversion to logarithms may avoid theproblem of rounding error. For example, logarithms of products ofnumbers become sums of logarithms of the same numbers, using theformulas log_(b)(AB)=log_(b)(A)+log_(b)(B) and log_(b)(A^(X))=Xlog_(b)(A). Match probabilities may be converted to match weights andback using, by way of non-limiting example, the following formulas:

W=−log(P); and   Equation 1

P=2^(−W).   Equation 2

In the above formulas, W denotes a weight and P denotes a probability.Note that, in general, match probabilities may be inversely related tothe match weights produced according to Equations 1 and 2. Thus, as aprobability grows, the associated weight, and therefore significance ofa match, decreases, and vice versa. The above formulas may be used forconverting numbers in general, not limited to match probabilities andmatch weights. One of ordinary skill in the art will understand how toconvert between standard form and logarithmic form and how to adapt theformulas herein in order to accommodate the different forms.

Match probabilities and match weights may be stored for later use. Forexample, these parameters may be stored in one or more lookup tables,alone or together with other relevant parameters. Alternately, or inaddition, these parameters may be stored in one or more fields added toeach record. By way of non-limiting example, field value weights may bestored in fields added to records in which the associated field valuesappear. The parameters may be updated with each iteration (per, forexample Section II) by replacing parameters from prior iterations or byadding newly generated parameters. In some embodiments, one or both offield value probabilities and field value weights may be stored infields appended to records, while one or both of field probabilities andfield weights may be stored in one or more lookup tables.

At block 115, a matching formula is selected according to the exemplaryembodiment. Such a matching formula may be, by way of non-limitingexample, as presented below in Equations 3-5. At block 120, a matchscore is calculated according to the matching formula selected at block115. Details of such calculations are discussed below in relation toEquations 3-5.

An exemplary technique for using field weights to make record linkingdecisions is discussed presently. Such decisions may take into accountsome or all of the fields common to the records. For example, alikelihood that two records reference the same individual may be scoredas:

$\begin{matrix}{{S\left( {r_{1},r_{2}} \right)} = {\sum\limits_{f}{p_{f}{w_{f}.}}}} & {{Equation}\mspace{14mu} 3}\end{matrix}$

In the above record matching formula, S(r₁, r₂) represents a scoreassociated with records r₁ and r₂, the sum may be over all fields fcommon to both r₁ and r₂, and each p_(f) may be a probability that thefield values of r₁ and r₂ match in field f. In an exemplary,non-limiting embodiment, if the field value in field f is non-null andidentical between records r₁ and r₂, then the corresponding probabilityp_(f) may be set equal to one, otherwise, it may be set equal to zero.In another exemplary, non-limiting embodiment, if the field values infield f are non-null and an exact or near match between records r₁ andr₂, then the corresponding probability p_(f) may be set equal to one,otherwise, it may be set equal to zero. Such embodiments areparticularly suitable for implementing the techniques of Sections IIIand IV, where a near match is determined according to certain distancefunctions. Alternate techniques for determining the probabilities p_(f)are disclosed in the First Generation Patents And Applications. Suchtechniques include those that assign nonzero probabilities p_(f) tofield values that are not exactly identical. Note that Equation 3 takesinto account all fields common to both r₁ and r₂. In Equation 3, eachw_(f) may be a field weight associated with field f. Techniques fordetermining these quantities are disclosed herein (e.g., as discussed indetail in reference to Equations 7, 11 and 15 below). Note that eachw_(f) may be a field weight as computed at any stage of an iteration;that is, each w_(f,v) as they appear in Equation 3 may be any ofw_(f)(1), w_(f)(2), etc. In this technique, knowledge of the commonfield values is not required, rather, knowledge that the field valuesmatch suffices. Note that, if a field value weight lookup table is largein comparison to a field weight lookup table, then computers cangenerally detect whether two fields contain identical field values andthen look up an associated field weight faster than they can detect thattwo fields contain the same field value and retrieve a field valueweight associated with the specific field value. Note further that usingfield weights produces accurate results for any two records, regardlessas to the contents of their fields.

The field value probabilities calculated by certain embodiments may beconverted to field value weights and used in making record linkingdecisions. Such decisions may take into account some or all of thefields common to the records. For example, a likelihood that two recordsreference the same individual may be scored as:

$\begin{matrix}{{S\left( {r_{1},r_{2}} \right)} = {\sum\limits_{f}{p_{f}{w_{f,v}.}}}} & {{Equation}\mspace{14mu} 4}\end{matrix}$

In the above record matching formula, S(r₁, r₂) represents a scoreassigned to records r₁ and r₂, the sum may be over all fields f commonto both r₁ and r₂, and each p_(f) may be a probability that the fieldvalues of r₁ and r₂ match in field f. In an exemplary, non-limitingembodiment, if the field value in field f is non-null and identicalbetween records r₁ and r₂, then the corresponding probability p_(f) maybe set equal to one, otherwise, it may be set equal to zero. In anotherexemplary, non-limiting embodiment, if the field values in field f arenon-null and an exact or near match between records r₁ and r₂, then thecorresponding probability p_(f) may be set equal to one, otherwise, itmay be set equal to zero. Such embodiments are particularly suitable forimplementing the techniques of Sections III and IV, where a near matchis determined according to certain distance functions, or according tothe techniques of Section X, where blended weights are used. Alternatetechniques for determining the probabilities p_(f) are disclosed in theFirst Generation Patents And Applications. Such techniques include thosethat assign nonzero probabilities p_(f) to field values that may not beexactly identical. Note that Equation 4 takes into account all fieldscommon to both r₁ and r₂. Unlike Equation 3, however, Equation 4 alsotakes into account the particular field values v appearing in each fieldf common to records r₁ and r₂. In Equation 4, each w_(f,v) may be afield value weight associated with field f and value v appearing infield f. Techniques for determining these quantities are disclosedherein (e.g., as discussed in detail in reference to Equations 8, 12 and16 below). Note that each w_(f,v) may be a field value weight ascomputed at any stage of an iteration; that is, each w_(f,v) as theyappear in Equation 4 may be any of w_(f,v)(1), w_(f,v)(2), etc. Usingfield value weights may require identifying the field values themselves.More particularly, using field value weights may involve using a look-uptable that is larger than a look-up table associated with the fieldweights. However, Equation 4 in general produces more accurate resultsin comparison with Equation 3, as Equation 4 is tailored to theparticular field values in the records being compared.

In some embodiments, a combination of field weights and field valueweights may be used in a matching formula. That is, Equations 3 and 4may be combined such that each term may contain the product of aprobability p_(f) and either a field value weight or a field weight.This technique may be useful, for example, when the field values areknow for some fields but not for others. For the fields in which thefield values are known, field value weights may be used, whereas for thefields in which the field values are not known, field weights may beused. Thus, Equations 3 and 4 may be mixed together.

In some embodiments, matching formulas as discussed in the FirstGeneration Patents And Applications may be used instead of, or inaddition to, the matching formulas presented herein. That is, thematching formulas presented in the First Generation Patents AndApplications in reference to EQs 1-4 therein may be used in anyembodiment disclosed in the present document that calls for a matchingformula.

More generally, a generic matching formula may be employed. Such ageneric matching formula may utilize multiple techniques presented inthis document. For purposes of illustration and by way of non-limitingexample, the generic matching formula may be expressed as:

$\begin{matrix}{{S\left( {r_{1},r_{2}} \right)} = {\sum\limits_{i = 1}^{I}{p_{i}{w_{i}.}}}} & {{Equation}\mspace{14mu} 5}\end{matrix}$

In Equation 5 above the index i may range over each field common to therecords under comparison, from one (1) to I. That is, the index term iserves to enumerate such common fields. In some embodiments, asdiscussed below, the range of i may vary depending on the results of aparticular comparison of a particular field using a particulartechnique. That is, some embodiments compare supplemental or proxyfields instead of an original field. In some embodiments, only termsthat correspond to the technique that yields the highest match weightmay be included in Equation 5. An overview of how Equation 5 may be usedis presented immediately below.

In some embodiments, comparing two records for the purpose of decidingwhether to link such records may proceed as follows. For each fieldcommon to the two records being compared, a term p_(i)w_(i) in Equation5 may be calculated as follows. The following discussion is relative toa field with index i common to the two records under comparison, withthe understanding that this process may be repeated for each commonfield in order to generate a corresponding term p_(i)w_(i) for each iless than I or as otherwise stated. First, a determination may be madeas to whether the field at issue (having index i) is accounted for in asupplemental field (e.g., with index j) according to a technique ofSection V. If so, then the comparison turns to the supplemental fieldwith index j. If there is an exact match between the records in thesupplemental field, then a term for that field may be included in thematching formula instead of a term for the original field with index i.Specifically, the term p_(j) may be set to one (1) or a value as setforth in the First Generation Patents And Applications, and the weightw_(j) may be set to any of, by way of non-limiting example, w_(j),w_(j,v), w_(j)(n) for any n, or w_(j,v)(n) for any n. (These terms aredefined in the sections following. Note that, in general and throughoutthis document, the term “w” with a subscript of, for example, i, j, k,or l may be interpreted as any match weight discussed herein.) If thereis not an exact match in the supplemental field (with index j), then thecomparison may proceed as follows, with the understanding that any ofthe weights may be modified by multiplication by a supplemental weight Was discussed in Section V.

If the particular field is not accounted for in a matching supplementalfield, then the field values with index i may be compared to detect ifthey are identical. If so, then a technique according to Section II maybe employed. Specifically, the term p_(i) may be set to one (1) or avalue as set forth in the First Generation Patents And Applications, andthe weight w_(i) may be set to any of, by way of non-limiting example,w_(i), w_(i,v), w_(i)(n) for any n, or w_(i,v)(n) for any n.

If the field values are not identical, the comparison may proceed asfollows. One or more of the techniques of Sections III, IV and X may beemployed to detect a near match in the field. For the technique ofSection III, an appropriate distance function D may be used to determinewhether the field values are a near match with respect to a distance d(e.g., whether D as applied to the field values produces a result nogreater than d). If so, p_(i) may be set to one (1) or a value as setforth in the First Generation Patents And Applications, and w_(i) may beset to w_(i,v,D,d), w_(i,D,d), w_(i,v,D,d)(n) for any n or w_(i,D,d)(n)for any n. In general, the weight having the least d may be used. Forthe technique of Section IV, instead of considering the original field(with index i) in which a non-exact match occurs, an alternate “proxy”field may be considered (with index k) in order to determine a nearmatch according to an appropriate distance function D. If there is anexact match in the proxy field, then p_(k) may be set to one (1) or avalue as set forth in the First Generation Patents And Applications, andw_(k) may be set to w_(k), w_(k,v), w_(k)(n) for any n or w_(k,v)(n) forany n. For the technique of Section X, p_(i) may be set to one (1) oranother value as set forth in the First Generation Patents AndApplications, and w_(i) may be set to any of the blended weights thatare discussed in Section X. For example, such blended weights may becomputed according to any of Equations 33-36, employing any of thetechniques discussed in relation to Tables X.5, X.7 and X.8.

Typically, one of the techniques from Sections III, IV and X willproduce a larger associated weight term w_(i) (for the techniques ofSections III or X) or w_(k)(for the technique of Section IV). Thetechnique that produces the largest weight term may be employed for thenear match term selection. Alternately, the technique that produces thelargest term p_(i)w_(i) (for the techniques of Sections III or X) orp_(k)w_(k) (for the technique of Section IV) may be used. The techniqueselection may occur on a term-by-term basis.

The comparisons discussed above may be repeated for each field common tothe records under comparison in order to calculate the score S(r₁=r₂) ofEquation 5. Once such a score is calculated, the following technique maybe used to determine whether the score is sufficiently large to justifylinking the records.

Thus, at block 125, the exemplary embodiment calculates a confidencelevel of the score produced by the matching formula. Exemplaryconfidence level calculation techniques are presented below in relationto Equation 6. If the confidence level is sufficiently high, the recordsare linked at block 130.

In certain embodiments, if a score S(r₁=r₂) exceeds a threshold, therecords may be linked. A technique for determining such a threshold isdisclosed presently. More particularly, a threshold may be calculatedas, by way of non-limiting example:

T=log(N)−log(1−P)−1.   Equation 6

In the Equation 6, the term N represents the total number of records inthe database for the purpose of the first iteration of a processdescribed in the sections below, or the total number of entityrepresentations for the second, third and subsequent iterations. Thus,the value of N may depend on the particular stage in the iteration inwhich Equation 6 is being used. Alternately, if the number of actualindividuals represented in the database is known (for example, if thedatabase is meant to reflect a known population, such as undergraduatesin a particular university), then that quantity may be used for N. Theterm P may be selected from the interval [0,1) (i.e., as a number bothgreater than or equal to zero and less than one) to establish aconfidence level. More particularly, if a score S(r₁, r₂) calculatedaccording to Equations 3, 4 or 5 with respect to two records r₁, r₂exceeds a threshold T calculated according to Equation 6, then theprobability that r₁ and r₂ refer to the same individual and should belinked is at least P. Note that P may be selected from the intervalbetween zero and one, inclusive, and may be converted to a percentage bymultiplication by 100. For each additional unit (i.e., 1) added to T,the quantity (1−P) halves (for embodiments that utilize log base two;for other bases, the quantity (1−P) may decrease as a power of thebase). In Equation 6, and throughout this disclosure, by way ofnon-limiting example, the log function has as its base two (2).Nevertheless, other bases may be used in embodiments of the presentinventions, such as, by way of non-limiting example, 2, 3⅓ or 10. Atable of thresholds computed for a variety of confidence levels appearsbelow.

P T   99% log(N) + 5.64  99.9% log(N) + 8.97 99.99% log(N) + 12.28

As is apparent from the table, the threshold computed using Equation 6may be dependent on the number of records in the database (for the firstiteration) or the number of entity representations in the database (forsubsequent iterations). If a score computed by Equations 3, 4 or 5exceeds a threshold T computed using Equation 6, then the probabilitythat the records should be linked is at least as great as the confidencelevel P. That is, the present technique allows records to be linked witha specified level of precision, i.e., a probability that a link betweenthe records will not be erroneous. Put another way, the presenttechnique allows records to be linked with a known probability (P) ofavoiding false positives.

As discussed elsewhere herein, after the first iteration, with eachiteration, the number of entity representations in the database may beexpected to decrease until it reaches a stable number. Accordingly, thevalue log(N) in Equation 6 may be reduced with each iteration (up to apoint) such that with each iteration, the threshold required for a givenfixed confidence level may be reduced.

The above described embodiments are exemplary only and are not intendedto limit the scope of the inventions disclosed herein.

II. Statistical Record Linkage Calibration at the Field and Field ValueLevels Without the Need for Human Interaction

In some embodiments, the techniques of this section provide one or moreweights, which may be used in a record matching formula (e.g., Equations3-5) to scale probabilities (e.g., p_(f) or p_(i)) that two recordscontain a particular field value in a particular field. In suchembodiments, a separate weight may be associated with each field value.Thus, certain embodiments associate to each field value a field valueprobability, which indicates the likelihood that a record or entityrepresentation chosen at random contains the associated field value.Such field value probabilities may be converted to field value weightsand used to make linking decisions as discussed above in Section I. Thatis, such field value weights may be used in conjunction with otherweights in making matching decisions, e.g., based on Equations 3-6above.

Certain embodiments associate a field probability to each field,independent of any particular field value. For a given field, theassociated probability may be computed as a weighted average of theprobabilities associated with each individual field value that may occurin the given field. Such computations are discussed in detail below inthe context of Equations 9, 13 and 17. The field probabilities soproduced may be converted to field weights and used to make linkingdecisions as discussed above in Section I. That is, such field valueweights may be used in conjunction with other weights in making matchingdecisions, e.g., based on Equations 3-6 above.

FIG. 2 is a flowchart depicting an exemplary embodiment of an inventionof Section II. An exemplary first iteration of the exemplary iterativematch probability and match weight producing embodiment is discussedpresently.

At block 205, the first iteration begins by calculating field valueprobabilities and field value weights. For every field value thatappears in any field in any record in the database, the first iterationproceeds by determining the number of records in the database thatinclude the field value in the associated field. That is, the firstiteration counts the number of records that include a particular fieldvalue in a particular field, and this counting may be performed forevery field value and field. At this point, every field value has anassociated count. These counts may be then divided by the total numberof non-null records in the database, yielding field value probabilities,each of which may be associated with a field value and the field inwhich the field value appears. That is, at the end of the firstiteration, each field value and the field in which it appears may beassociated with a field value probability, which may be calculated asthe number of records that include the field value in the respectivefield divided by the total number of records. For a given field f andfield value v, the associated field value probability may be calculatedas, by way of non-limiting example:

$\begin{matrix}{{p_{f,v}(1)} = {\frac{c_{f,v}}{c}.}} & {{Equation}\mspace{14mu} 7}\end{matrix}$

In Equation 7, the term P_(f,v)(1) represents the first iteration fieldvalue probability associated with field f and field value v appearing infield f. The term c_(f,v) represents the number of records that includefield value v in field f, and the term c represents the total number ofrecords in the database. Accordingly, a given field value probabilityproduced by the first iteration may be a probability that a recordrandomly chosen from the database contains the given field value in itsassociated field. The field value probabilities may be converted tofield value weights according to, by way of non-limiting example:

w _(f,v)(1)−log p _(f,v)(1).   Equation 8

Thus, at the end of the first iteration, each field value and the fieldin which it appears may be associated with a field value weight, each ofwhich may be calculated from a corresponding field value probability.

The field value probabilities and field value weights may be stored forlater use. For example, these parameters may be stored in a lookuptable, alone or together with other relevant parameters. Alternately, orin addition, these parameters may be stored in one or more fields addedto each record. By way of non-limiting example, field value weights maybe stored in fields added to records in which the associated fieldvalues appear. The parameters may be updated with each iteration byreplacing parameters from prior iterations or by adding newly generatedparameters. In some embodiments, one or both of field valueprobabilities and field value weights may be stored in fields appendedto records, while one or both of field probabilities and field weightsmay be stored in one or more lookup tables.

Note that computation of each field value probability (or field valueweight) may occur once for each distinct field value. In subsequentconsiderations of records with particular field values, the associatedfield value probability (or field value weight) may be retrieved from astorage location, such as a lookup table or a record itself.

At block 210, the exemplary first iteration may also produce fieldprobabilities and field weights for every field that appears in anyrecord in the database. The field probabilities may be calculated asweighted sums of field value probabilities. More particularly, for agiven field f, the associated first iteration field probability may becalculated as, by way of non-limiting example:

$\begin{matrix}{{p_{f}(1)} = {\sum\limits_{v}{\left( {p_{f,v}(1)} \right)^{2}.}}} & {{Equation}\mspace{14mu} 9}\end{matrix}$

In Equation 9, p_(f)(1) represents the first iteration field probabilityassociated with field f. The sum may be over all field values v thatappear in any record in field f. Each term p_(f,v)(1) represents thefirst iteration field value probability associated with field value v offield f. The field probabilities may represent the likelihood that tworecords chosen at random will share a common field value in theassociated field. Note that Equation 9 may be considered as a weightedsum, where the sum may be over all field value probabilities and theweights themselves are also field value probabilities (hence the squaredterm). Note further that Equation 9 may be considered as a weightedaverage of field value probabilities. The field probabilities may beconverted to field weights according to, by way of non-limiting example:

w _(f)(1)=−log p _(f)(1).   Equation 10

Thus, the first iteration may calculate field probabilities and fieldweights for every field in every record in the database according toEquations 9 and 10, concluding the first iteration. The fieldprobabilities and field weights may be stored for later use as discussedabove.

At block 215, between the first iteration and the second iteration, thedatabase may undergo a preliminary linking operation, which may be basedon the match weights generated by the first iteration. Such an exemplarylinking operation is discussed presently. Each record may be compared toevery other record in the database. Each such comparison may result in alink between the compared pair of records, depending on the results ofthe comparison. (In some embodiments, every record may be compared witha subset of other records in the database. Such a subset may begenerated using blocking criteria as disclosed elsewhere herein.) In theexemplary linking operation, given records r₁ and r₂ and faced with adecision to link them, Equations 3, 4 or 5 may be used to calculate ascore that the records reference the same individual. If the scoreexceeds a threshold, the records r₁ and r₂ may be linked. Such athreshold may be determined as discussed in relation to Equation 6.

In general, the actual linking of two records may be performed, by wayof non-limiting example, as discussed in the First Generation PatentsAnd Applications, e.g., by inserting a Definitive Identifier (“DID”) inan appropriate field of both records. Note that the linking decision maybe made for every pair of records in the database, or for pairs ofrecords generated by blocking criteria. The result of the preliminarylinking operation may be that the database now contains entityrepresentations, that is, multiple sets of linked records, where eachsuch linked set is meant to contain records that correspond to the sameindividual.

At block 220, intermediate operations may be performed. Exemplary suchoperations (e.g., transitional linking, propagation, delinking) arediscussed presently.

The database may undergo a transitional linking process between thefirst iteration and the second iteration. Examples of suitabletransitional linking processes are discussed in the First GenerationPatents And Patent Applications. Another example of a suitabletransitional linking process is disclosed in Section VII below. Thetransitional linking process may occur at any stage between iterations,e.g., before or after preliminary linking operation 215.

The database may undergo a propagation operation between the firstiteration and the second iteration. Such a propagation operation mayinsert missing field values in recently linked records. For example, ifthe first iteration results in a first record and a second record beinglinked, and the first record contains a null value in a field in whichthe second record includes a non-null field value, then the non-nullfield value may be propagated to the first record.

Likewise, if the second record contains a null field value in a field inwhich the first record contains a non-null field value, than that valuemay be propagated to the second record. A specific example follows.Consider records r₁ and r₂ reflected below.

Street Record First Name Last Name Address SSN Gender r₁ John Smith 123Fake St. 999-99-999 r₂ John Smith 123 Fake St. Male

If the first iteration results in a link between these records, then theSSN of the first record may be propagated to the second record and theGender of the second record may be propagated to the first record. Theresulting records after the propagation step may appear as follows:

Street Record First Name Last Name Address SSN Gender r₁ John Smith 123Fake St. 999-99-999 Male r₂ John Smith 123 Fake St. 999-99-999 Male

In the above table, the field values propagated between linked recordsare italicized for purposes of illustration.

Note that it may be possible for two records linked in the same entityrepresentation to have different field values in the same field. Forpurposes of the propagation operation, mechanisms for selecting thevalue to propagate to records having null in the associated field arediscussed presently. In some embodiments, the field value that occursmost frequently in a given field in records linked to the same entityrepresentation may be propagated to records linked to the same entityrepresentation that contain a null value in the given field. In the casewhere two or more field values occur with the same frequency in a givenfield of records linked to the same entity representation, the fieldvalue with the most information (highest specificity) may be selectedfor propagation.

The database may undergo a delinking operation between the firstiteration and the second iteration. Such a delinking operation maydelink records that were incorrectly linked by the preliminary linkingoperation. Exemplary delinking operations are disclosed in the FirstGeneration Patents And Applications.

Thus blocks 205, 210 and 215 represent a first iteration of theexemplary embodiment under discussion. The exemplary embodiment mayfurther include block 220 in the first iteration. Subsequent iterations,as explained below, may include blocks 225, 230, 235, and may alsoinclude block 240.

A second exemplary iteration of the exemplary process is discussedpresently. Like the first iteration, the second iteration produces matchprobabilities and match weights. However, the match probabilities andmatch weights produced by the second iteration may generally be moreaccurate than those produced by the first iteration. Furthermore, asdiscussed in detail below, iterations after the first iteration may takea number of entity representations into account.

At block 225, the second iteration begins by calculating field valueprobabilities and field value weights. After the first iteration, thedatabase contains sets of linked records in the form of entityrepresentations. The second iteration begins by counting, for eachnon-null field value that appears in an associated field in any recordin the database, the number of entity representations that contain atleast one record with that non-null field value in the associated field.Thus, the second iteration begins by associating to each field value thenumber of entity representations that include a record including suchfield value. Each of these counts may be then divided by the totalnumber of entity representations in the database, resulting in fieldvalue probabilities. Thus, for a given field f and field value v, theassociated field value probability may be calculated as, by way ofnon-limiting example:

$\begin{matrix}{{p_{f,v}(2)} = {\frac{k_{f,v}}{k}.}} & {{Equation}\mspace{14mu} 11}\end{matrix}$

In Equation 11, the term p_(f,v)(2) represents the second iterationfield value probability associated with field f and field value vappearing in field f. The term k_(f,v) represents the number of entityrepresentations that include a record that includes field value v infield f, and the term k represents the total number of entityrepresentations in the database. Thus, the second iteration produces,for each field value, an associated field value probability, which maybe calculated as the ratio of the number of entity representationscontaining a record containing the field value to the total number ofentity representations in the database. Accordingly, a given field valueprobability produced by the second iteration may be a probability thatan entity representation randomly chosen from the database contains arecord with the given field value in its associated field. The seconditeration field value probabilities may be converted to field valueweights according to, by way of non-limiting example:

w _(f,v)(2)−log p _(f,v)(2).   Equation 12

Thus, at the end of the second iteration, each field value and the fieldin which it appears may be associated with a field value weight, each ofwhich may be calculated from a corresponding field value probability.The field value probabilities and field value weights may be stored forlater use as discussed above.

At block 230, the second iteration produces field probabilities andfield weights associated with each field appearing in any record in thedatabase. The field probabilities may be calculated as weighted sums ofthe field value probabilities produced by the second iteration. Moreparticularly, for a given field f, the associated field probability maybe calculated as, by way of non-limiting example:

$\begin{matrix}{{p_{f}(2)} = {\sum\limits_{v}{\left( {p_{f,v}(2)} \right)^{2}.}}} & {{Equation}\mspace{14mu} 13}\end{matrix}$

In the Equation 13, p_(f)(2) represents the second iteration fieldprobability associated with field f. The sum may be over all fieldvalues v that appear in any record in field f. Each term p_(f,v)(2)represents the second iteration field value probability associated withfield value v of field f. Each field probability may represent theprobability that two entity representations chosen at random willcontain records that share a common field value in the associated field.Note that Equation 13 may be considered as a weighted sum, where the summay be over all field value probabilities and the weights themselves arealso field value probabilities (hence the squared term). Note furtherthat Equation 13 may be considered as a weighted average of field valueprobabilities. The field probabilities may be converted to field weightsaccording to, by way of non-limiting example:

w _(f)(2)−log p _(f)(2).   Equation 14

Thus, the second iteration may calculate field probabilities and fieldweights for every field in every record in the database according toEquations 13 and 14, concluding the second iteration. The fieldprobabilities and field weights may be stored for later use as discussedabove.

At block 235, between the second iteration and the third iteration, thedatabase undergoes a linking operation, which may be based on the matchweights generated by the second iteration. The linking operation betweenthe second and third iterations may be essentially identical to thelinking operation between the first iteration and the second iteration.Thus, each given record may be compared to every other record in thedatabase (or to a set of records generated by blocking criteria) towhich the given record is not already linked, and each such comparisonmay result in a link between the compared records and, therefore, thecorresponding pair of entity representations. Each such comparison mayresult in a link between the compared pair of records, depending on theresults of the comparison. (In some embodiments, every record may becompared with a subset of other records in the database. Such a subsetmay be generated using blocking criteria as disclosed elsewhere herein.)In the exemplary linking operation, given records r₁ and r₂ and facedwith a decision to link them, Equations 3, 4 or 5 may be used tocalculate a score that the records reference the same individual. If thescore exceeds a threshold, the records r₁ and r₂ may be linked. Such athreshold may be determined as discussed in relation to Equation 6.Linking these records links, in turn, the entity representations towhich the records may be linked.

It is likely that once the linking operation occurs between the seconditeration and the third iteration, the number of unique entityrepresentations in the database may be reduced in comparison with thenumber that existed after the second iteration.

At block 240, intermediate operations may be performed. Exemplary suchoperations (e.g., transitional linking, propagation, delinking) arediscussed presently.

The database may undergo a transitional linking process between thesecond iteration and the third iteration. Examples of suitabletransitional linking processes are discussed in the First GenerationPatents And Patent Applications. Another example of a suitabletransitional linking process is presented in Section VII. Thetransitional linking process may occur at any stage between iteration,e.g., before or after linking operation 235.

The database may undergo a propagation operation between the seconditeration and third iteration. The propagation operation may beessentially the same as the propagation operation that may occur betweenthe first iteration and the second iteration. That is, null field valuesin a first record may be replaced by non-null field values taken from asecond record to which the first record may be linked. Likewise, nullfield values in the second record may be replaced by non-null fieldvalues taken from the first record.

The database may undergo a delinking operation between the seconditeration and the third iteration. Such a delinking operation may delinkrecords that were incorrectly linked by the preliminary linkingoperation. Exemplary delinking operations are disclosed in the FirstGeneration Patents And Applications.

Block 245 indicates that one or more of blocks 225, 230, 235 and 240 maybe iterated. Third, fourth and subsequent iterations of the exemplaryprocess for generating match probabilities proceeds in a manner similarto that of the second iteration. Thus, the third iteration andsubsequent iterations each produce field value probabilities, fieldvalue weights, field probabilities, and field weights. Moreover, thematch probabilities and match weights produced by each successiveiteration may generally be more accurate than those produced by theprior iteration.

In the third, fourth and subsequent iterations of the exemplary process,field value probabilities and field value weights may be calculated inthe same manner as those of the second iteration (block 225). Namely,each field value probability may be calculated using a ratio of thenumber of entity representations containing a record containing thefield value to the total number of entity representations in thedatabase. Thus, for a given field f and field value v, the associatedfield value probability may be calculated as, by way of non-limitingexample:

$\begin{matrix}{{p_{f,v}(n)} = {\frac{k_{f,v}}{k}.}} & {{Equation}\mspace{14mu} 15}\end{matrix}$

In Equation 15, the term p_(f,v)(n) represents the n-th iteration fieldvalue probability associated with field f and field value v appearing infield f. The term k_(f,v) represents the number of entityrepresentations (existing at the time the n-th iteration is executed)that include a record that includes field value v in field f, and theterm k represents the total number of entity representations in thedatabase (again, existing at the time the n-th iteration is executed).Thus, the n-th iteration produces, for each field value, an associatedfield value probability, which may be calculated as the ratio of thenumber of entity representations containing a record containing thefield value to the total number of entity representations in thedatabase. Accordingly, a given field value probability produced by thesecond iteration may be a probability that an entity representationrandomly chosen from the database contains a record with the given fieldvalue in its associated field. The n-th iteration field valueprobabilities may be converted to field value weights according to, byway of non-limiting example:

w _(f,v)(n)=−log p _(f,v)(n)   Equation 16

Thus, at the end of the n-th iteration, each field value and the fieldin which it appears may be associated with a field value weight, each ofwhich may be calculated from a corresponding field value probability.The field value probabilities and field value weights may be stored forlater use as discussed above.

The field probabilities and field weights produced by the third, fourthand subsequent iterations may be calculated in essentially the samemanner as in the second iteration (block 230). These may be calculatedas weighted sums of the field value probabilities produced by the n-thiteration. More particularly, for a given field f, the associated fieldprobability may be calculated as, by way of non-limiting example:

$\begin{matrix}{{p_{f}(n)} = {\sum\limits_{v}\; {\left( {p_{f,v}(n)} \right)^{2}.}}} & {{Equation}\mspace{14mu} 17}\end{matrix}$

In the Equation 17, p_(f)(n) represents the n-th iteration fieldprobability associated with field f. The sum may be over all fieldvalues v that appear in any record in field f. Each term p_(f,v)(n)represents the n-th iteration field value probability associated withfield value v of field f. Note that Equation 17 may be considered as aweighted sum, where the sum may be over all field value probabilitiesand the weights themselves are also field value probabilities (hence thesquared term). Each field probability may represent the probability thattwo entity representations chosen at random will contain records thatshare a common field value in the associated field. Note further thatEquation 17 may be considered as a weighted average of field valueprobabilities. The field probabilities may be converted to field weightsaccording to, by way of non-limiting example:

w _(f)(n)=−log p _(f)(n).   Equation 18

Thus, the n-th iteration may calculate field probabilities and fieldweights for every field in every record in the database according toEquations 17 and 18, concluding the n-th iteration. Note that the termsin Equations 17 and 18 for the n-th iteration may be determined by theprior (i.e., (n−1)-th) iteration and subsequent linking, delinking andpropagation processes. The field probabilities and field weights may bestored for later use as discussed above.

Subsequent intermediate operations (e.g., linking, transition linking,propagation and delinking blocks, block 240) may follow each iteration.It may be expected that each iteration produces more accurate matchprobabilities (and match weights), converging to match probabilities(respectively, match weights) that may be highly accurate after asuitable number of iterations. Note that the terms w_(f) and w_(f,v)appearing in Equations 3, 4 and 5 may represent the results of anyiteration. That is, the terms w_(f), w_(f,v) and w_(i) appearing inEquations 3, 4 and 5 may represent w_(f)(n) or w_(f,v)(n) for any n. Itmay also be expected that each iteration results in fewer entityrepresentations in the database, converging to a stable number of entityrepresentations after a suitable number of iterations.

The iteration may halt after any number of iterations after any ofblocks 225, 230, 235 or 240. At block 250, the match weights and matchprobabilities may be used to link records as discussed elsewhere herein.

Note that field weights produced according to certain embodiments of thepresent invention have several useful properties. More particularly,field weights may be calculated according to Equations 10, 14 and 18,and such weights have several useful properties. For the purposes ofthis disclosure, the term “cohort” means the set of entity records (forthe first iteration) or entity representations (for second andsubsequent iterations) that share a common field value. Thus, forexample, after the first iteration, the collection of all entityrepresentations that have “John” in the First Name field of an includedrecord may be considered one cohort, and the collection of all entityrepresentations that have “Mary” in the First Name field of an includedrecord may be considered another cohort. Thus, a particular field mayhave many different cohorts associated with it. For example, the FirstName field may be associated with cohorts that correspond to each uniquefirst name that appears in any record in the database. Certainembodiments of the present invention produce field weights with theproperty that, the larger the weight associated with a field, the moresignificance may be accorded a match between two records in that field.Field weights calculated according to certain embodiments of the presentinvention may accord significance in a manner that takes into accountboth the number of cohorts and how the field values are distributedamong the cohorts.

An example may help illustrate. Consider the First Name field, andsuppose for purposes of illustration that there are exactly four firstname field cohorts in the database: John, Mary, Dick and Jane. Supposethat the four cohorts are distributed exactly evenly; that is, eachcohort occupies exactly 25% of the records (or the entityrepresentations). Then the field probability for the first name fieldmay be calculated according to Equations 9, 13 or 17 as:

(¼)²+(¼)²+(¼)²+(¼)²=¼.

The corresponding field weight may be calculated according to Equations10, 14 or 18 as:

−log(¼)=2

Thus, a match between two records at the first name field may beaccorded a weight, or significance, of two.

Now consider a database in which there are four First Name cohorts thatare distributed unevenly. By way of example, consider a database inwhich 97% of the records (or entity representations) fall within the“John” first-name cohort, 1% of the records (or entity representations)fall within the “Mary” cohort, 1% fall within the “Dick” cohort, and 1%fall within the “Jane” cohort. Then the field probability for the firstname field may be calculated according to Equations 9, 13 or 17 as:

(0.97)²+(0.01)²+(0.01)²+(0.01)²=0.941.

The corresponding field weight may be calculated according to Equations10, 14 or 18 as:

−log(0.9412)=0.0874.

Accordingly, a match between two records at the first name field may beaccorded a weight, or significance, of 0.0874. Note that thesignificance accorded a match in this example is very close to zero,whereas a significance accorded a match in the previous example is muchmore significant. Thus, certain embodiments of the present inventionproduce matching formula weights that take into account both the numberand the distribution of the cohorts in each field.

Note that in the above example, the field value weight associated withthe cohort “John” (namely, 0.0439) is much less than the field valueweight associated with any of the cohorts “Mary,” “Dick,” or “Jane”(namely 6.64).

Advantages of the iterative technique disclosed herein over the priorart include the following. Certain prior art techniques, such as thosedisclosed in U.S. Pat. No. 6,523,019 to Borthwick, require humanintervention and manual entry or examination of linking decisions. Forexample, U.S. Pat. No. 6,523,019 to Borthwick requires a training stepin which a human operator is required to analyze and make linkingdecisions for large sets of data. In contrast, certain embodiments ofthe present technique may be executed without any human decision making.While embodiments of the present invention may include manual entry orexamination of linking decisions, these features are not required. Yetembodiments of the present invention still provide accurate matchingformula coefficients or weights. Other advantages include improvedaccuracy and the ability to quickly recalculate match probabilities uponadditional records being added to the database. In such instances, onceadditional records are added, the process may simply be iterated one ormore times in order to assimilate the additional records into entityrepresentations as appropriate.

According to an exemplary embodiment of a technique of this section, amethod of utilizing a record matching formula weight, where the recordmatching formula weight is specific to a particular field value, ispresented. The embodiment calculates a first probability that a recordin the database includes the particular field value. The embodimentlinks records in the database based at least in part on the firstprobability, whereby a plurality of entity representations aregenerated. The embodiment calculates a second probability that an entityrepresentation in the database includes the particular field value. Theembodiment links entity representations in the database based at leastin part on the second probability. And the embodiment allows a user toretrieve information from at least one record in the database.

Various optional features for the above-described exemplary embodimentinclude the following. The embodiment may further iterate calculating asecond probability and linking entity representations at least onceprior to the retrieving. The embodiment may further include calculatinga probability that two records match using the record matching formula,where the record matching formula includes a weighted sum ofprobabilities that two records match, where the weights include thesecond probability.

According to an exemplary embodiment of a technique of this section, amethod of using a record matching formula weight, where the recordmatching formula weight is specific to a particular field andindependent of any particular field value in the particular field, ispresented. The embodiment calculates a plurality of first probabilities,each of the plurality of first probabilities reflecting a likelihoodthat a record in the database includes a particular field value. Theembodiment calculates a first weight comprising a weighted sum of thefirst probabilities. The embodiment links records in the database basedat least in part on the first weight, whereby a plurality of entityrepresentations are generated. The embodiment calculates a secondplurality of probabilities, each of the second plurality ofprobabilities reflecting a likelihood that an entity representation inthe database includes a particular field value. The embodimentcalculates a second weight comprising a weighted sum of the secondprobabilities. The embodiment links entity representations in thedatabase based at least in part on the second weight. And the embodimentallows a user to retrieve information from at least one record in thedatabase.

The above described embodiments are exemplary only and are not intendedto limit the scope of the inventions disclosed herein.

III. Statistical Record Linkage Calibration for Reflexive and SymmetricDistance Measures at the Field and Field Value Levels Without the Needfor Human Interaction

Embodiments of this technique may be implemented in its own iterativeprocess or incorporated into the iterative process described above inSection II.

In some embodiments, the techniques of this section provide one or moreweights, which may be used in a record matching formula (e.g., Equations3-5) to scale probabilities (e.g., p_(f) or p_(i)) that two recordscontain nearly matching field values in a particular field. Whether twofield values qualify as nearly matching may be determined in part by areflexive and symmetric distance function and a specified distancewithin which the two field values lie as determined by the function.

Thus, some embodiments provide field value probabilities associated withnear matches. For a record that contains a particular field value in aparticular field, certain embodiments provide a probability that arecord or entity representation chosen at random contains a field valuein the particular field that lies within a specified distance of theparticular field value. In such embodiments, a separate probability maybe associated with one or more distances and each field value. That is,certain embodiments of the present invention associate to each givenfield value and each chosen distance a probability that a record (orentity representation) chosen at random contains (respectively, containsa record that contains) a field value in the associated common fieldthat lies within the chosen distance of the given field value, where thedistance between field values may be determined by the reflexive andsymmetric distance function. Such field value probabilities may beconverted to field value weights and used to make linking decisions asdiscussed above in Section I. That is, such field value weights may beused in conjunction with other weights in making matching decisions,e.g., based on Equations 3-6 above.

Certain embodiments associate a probability to each field and selecteddistance, independent of any particular field value. For a given fieldand distance, the associated probability may be computed as a weightedaverage of the probabilities associated with the distance and eachindividual field value that may occur in the given field. Suchcomputations are discussed in detail below in the context of Equations21 and 25. The field probabilities calculated by certain embodiments maybe converted to field weights and used in making record linkingdecisions. Such decisions may take into account some or all of thefields common to the records. In this technique, knowledge of the commonfield values may not be required. Further, this technique producesaccurate results for any two records, regardless as to the contents oftheir fields.

FIG. 3 is a flowchart depicting an exemplary embodiment of an inventionof Section III. An exemplary first iteration of the exemplary iterativematch value match weight producing embodiment is discussed presently.

At block 300 a distance function is selected. Certain embodiments of thepresent invention allow for a variety of measures of what constitutes anear match of field values. In some embodiments, an edit distancefunction may be used. Such functions measure how many discrete editswould be required to change one field value into another field value.There are several types of edit distance metrics, including, by way ofnon-limiting example, Hamming distance, Levenshtein distance,Damerau-Levenshtein distance, Jaro-Winkler distance, Wagner-Fischerdistance, Ukkonen distance and Hirshberg distance. By way ofillustration, the Hamming distance between field values “disk” and“disc” is one (1), as one substitution would be required to transformone field value to the other. For purposes of illustration only, Hammingdistance will be used to illustrate the relevant properties of editdistance functions.

Importantly, the present invention is not limited to edit distancefunctions. Indeed, any function that is symmetric and reflexive maysuffice. The edit distance functions benefit from both properties. Moreparticularly, a function is symmetric if reversing the order ofarguments produces the same result. Edit distance has this property,because, for example, one substitution is required to transform “disk”to “disc”, and one substitution is required to transform “disc” to“disk”. Formally, if D(·, ·) is an edit distance function, then D(v₁,v₂)=D(v₂, v₁). The edit distance functions are also reflexive. That is,these functions output a distance of zero when two identical fieldvalues are compared. Thus, for example, no substitutions are required totransform the field value “database” to the field value “database”.Formally, if D(·, ·) is an edit distance function, then D(v₁, v₁)=0.Thus, any symmetric and reflexive function that compares field valuesand outputs a distance between the field values or a probability thatthe field values match may be used. For the remainder of this section,the term D denotes a function, not limited to edit distance functions,with the appropriate properties. Note that unary functions or binaryfunctions may be used with the present technique. Further, if theselected function is transitive in addition to being reflexive andsymmetric, then the techniques of Section IV may be used instead of thetechniques of the present section.

At block 305, exemplary first iteration continues by calculating fieldvalue probabilities and field value weights associated with a givendistance function D. For every given non-null field value that appearsin any field in any record in the database, the first iteration proceedsby determining the number of records in the database that include afield value in the associated field that is within a fixed distance ofthe given field value. This step may be performed for a number ofdifferent distances (e.g., for a edit distance function, the techniquemay be performed for distances of 1, 2, 3, etc.). That is, for eachgiven field and associated given field value, the first iteration countsthe number of records containing other field values that lie within afixed distance of the given field value. This process may be performedfor every field value.

At this point, every field value and distance has an associated recordcount. These counts may then be divided by the total number of recordscontaining non-null field values in the associated field, yielding fieldvalue probabilities, each of which may be associated with a field value,the field in which the field value appears, and a distance. That is, atthe end of the first iteration, each field value, the field in which itappears, and a given distance may be associated with a field valueprobability. Each field value probability may be calculated as thenumber of records containing fields with field values that lie withinthe given distance of the field value, divided by the number of recordscontaining non-null field values that appear in the field. For a givenfield f, field value v, function D, and distance d, the associated fieldvalue probability may be calculated as, by way of non-limiting example:

$\begin{matrix}{{p_{f,v,D,d}(1)} = {\frac{c_{f,v,D,d}}{c}.}} & {{Equation}\mspace{14mu} 19}\end{matrix}$

In Equation 19, the term p_(f,v,D,d)(1) represents the first iterationfield value probability associated with field f, field value v appearingin field f, the distance d and the function D. The term c_(f,v,D,d)denotes the number of records that contain a field value in field f thatis within distance d from field value v as measured by the function D.That is, c_(f,v,D,d) represents the number of records that contain afield value v′ in field f such that D(v,v′)≦d. The term c represents thenumber of records that contain a non-null field value in field f. Thus,the quotient on the right hand side of Equation 19 reflects a proportionof records that include a field value that lies within distance d of theparticular field value v. Accordingly, a given field value probabilityproduced by the first iteration of this technique may be a probabilitythat a record randomly chosen from the database contains a field valuein its associated field that lies within the distance d of a given fieldvalue. To calculate p_(f,v,D,d)(1), it suffices to calculate D(v,v′) foreach distinct v′ that appears in field f in any record in the database,and then, for values that lie within d of v, multiply by the number ofoccurrences of each (in records) and sum the results to get c_(f,v,D,d).The field value probabilities may be converted to field value weightsaccording to, by way of non-limiting example:

w _(f,v,D,d)(1)=−log p _(f,v,D,d)(1).   Equation 20

Thus, at the end of the first iteration, for at least one distance d,each field value v and the field in which it appears may be associatedwith a field value weight, each of which may be calculated from acorresponding field value probability.

The field value probabilities and field value weights may be stored forlater use. For example, these parameters may be stored in a lookuptable, alone or together with other relevant parameters such as theassociated distance(s). Alternately, or in addition, these parametersmay be stored in one or more fields added to each record. By way ofnon-limiting example, field value weights may be stored in fields addedto records in which the associated field values appear. The parametersmay be updated with each iteration by replacing parameters from prioriterations or by adding newly generated parameters. In some embodiments,one or both of field value probabilities and field value weights may bestored in fields appended to records, while one or both of fieldprobabilities and field weights may be stored in one or more lookuptables.

At block 310, the first iteration produces field probabilities and fieldweights for every field that appears in any record in the database andfor at least one distance. The field probabilities may be calculated asweighted sums of field value probabilities. More particularly, for agiven field f, the associated first iteration field probability may becalculated as, by way of non-limiting example:

$\begin{matrix}{{p_{f,D,d}(1)} = {\sum\limits_{v}\; {\left( {p_{f,v,D,d}(1)} \right)^{2}.}}} & {{Equation}\mspace{14mu} 21}\end{matrix}$

In the Equation 21, p_(f,v,D,d)(1) represents the first iteration fieldprobability associated with field f, distance d and function D. The summay be over all field values v that appear in any record in field f.Note that Equation 21 may be considered as a weighted sum, where the summay be over all field value probabilities and the weights themselves arealso field value probabilities (hence the squared term). Note furtherthat Equation 21 may be considered as a weighted average of field valueprobabilities. The field probabilities may be converted to field weightsaccording to, by way of non-limiting example:

w _(f,D,d)(1)=−log p _(f,D,d)(1).   Equation 22

Thus, the first iteration may calculate, according to Equations 21 and22, field probabilities and field weights for every field in everyrecord in the database, for at least one distance d, and with respect toa given reflexive and symmetric function D, concluding the firstiteration. The field probabilities and field weights may be stored forlater use as discussed above.

At block 315, between the first iteration and the second iteration, thedatabase undergoes a preliminary linking operation, which may be basedon the match weights generated by the first iteration. This linkingoperation may be essentially the same as the preliminary linkingoperation discussed above in Section I. The result of the preliminarylinking operation may be that, after the first iteration, the databasecontains entity representations, that is, multiple sets of linkedrecords, where each such linked set is meant to contain records thatcorrespond to the same individual.

At block 320, intermediate operations may be performed. Exemplary suchoperations (e.g., transitional linking, propagation, delinking) arediscussed presently.

The database may undergo a transition linking process between the firstiteration and the second iteration. The transition linking process maybe essentially the same as the transition linking process discussedabove in Section II. The transitional linking process may occur at anystage between iterations, e.g., before or after preliminary linkingoperation 315.

The database may undergo a propagation operation between the firstiteration and the second iteration. The propagation operation may beessentially the same as the propagation operation discussed above inSection II.

The database may undergo a delinking operation between the firstiteration and the second iteration. The delinking operation may beessentially the same as the delinking operation discussed above inSection II.

Thus, blocks 305, 310 and 315 represent a first iteration of theexemplary embodiment under discussion. The exemplary embodiment mayfurther include block 320 in the first iteration. Subsequent iterations,as explained below, may include blocks 325, 330, 335, and may alsoinclude block 340.

A technique for second, third and subsequent exemplary iterations of theexemplary process is discussed presently. Like the first iteration, thesecond iteration produces match probabilities and match weights.However, the match probabilities and match weights produced by thesecond iteration may generally be more accurate than those produced bythe first iteration. After the first iteration, the database containssets of linked records in the form of entity representations.

At block 325, the second (and subsequent) iteration begins bycalculating field value probabilities and field value weights. Thesecond (and subsequent) iteration may proceed by counting, for eachgiven non-null field value that appears in an associated field in anyrecord in the database, for at least one distance, and for a reflexivesymmetric function, the number of entity representations that contain atleast one record with a field value in the associated field that iswithin the distance of the given non-null field value as measured by thefunction. Thus, the second iteration begins by associating to each givenfield value the number of entity representations that include a recordincluding a field value that is “near” the given field value as measuredby the function. The function D and distance d define, by way ofnon-limiting example, what is meant by “near.” Each of these counts maythen be divided by the total number of entity representations in thedatabase that contain a record with a non-null field value in theassociated field, resulting in field value probabilities. Thus, for agiven distance d, function D, and a given field f and field value v, theassociated field value probability may be calculated as, by way ofnon-limiting example:

$\begin{matrix}{{p_{f,v,D,d}(n)} = {\frac{k_{f,v,D,d}}{k}.}} & {{Equation}\mspace{14mu} 23}\end{matrix}$

In Equation 23, the term p_(f,v,D,d)(n) represents the n-th iterationfield value probability associated with field f, field value v appearingin field f, the distance d and the function D. The term k_(f,v,D,d)denotes the number of entity representations containing records thatcontain a field value in field f that is within distance d from fieldvalue v as measured by the function D. That is, k_(f,v,D,d) representsthe number of entity representations that contain records that contain afield value v′ in field f such that D(v,v′)≦d. The term k represents thenumber of entity representations containing records that contain anon-null field value in field f. Thus, the quotient on the right handside of Equation 23 reflects a proportion of entity representations thatinclude a record containing a field value that lies within distance d ofthe particular field value v. Thus, the n-th iteration produces, foreach field value v, an associated field value probability, which may becalculated as the ratio of (1) the number of entity representationscontaining a record containing a field value v′ in field f that iswithin distance d of field value v to (2) the total number of entityrepresentations containing records with non-null field values in field fin the database.

Accordingly, a field value probability associated with a given fieldvalue and produced by the n-th iteration may be a probability that anentity representation randomly chosen from the database contains arecord with a field value in the associated field that lies withindistance d of the given field value. To calculate p_(f,v,D,d)(n), itsuffices to calculate D(v,v′) for each distinct v′ that appears in fieldf in any record in the database, and then, for values that lie within dof v, multiply by the number of occurrences of each (in entityrepresentations) and sum the results to get k_(f,v,D,d).

The n-th iteration field value probabilities may be converted to fieldvalue weights according to, by way of non-limiting example:

w _(f,v,D,d)(n)=log p _(f,v,D,d)(n)   Equation 24

Thus, at the end of the n-th iteration, each field value and the fieldin which it appears may be associated with a field value weight, each ofwhich may be calculated from a corresponding field value probability.The field value probabilities and field value weights may be stored forlater use as discussed above.

At block 330, the n-th iteration may produce field probabilitiesassociated with each field appearing in any record in the database.These may be calculated as weighted sums of the field valueprobabilities produced by the n-th iteration. More particularly, for agiven field f, function D and distance d, the associated fieldprobability may be calculated as, by way of non-limiting example:

$\begin{matrix}{{p_{f,D,d}(n)} = {\sum\limits_{v}\; {\left( {p_{f,v,D,d}(n)} \right)^{2}.}}} & {{Equation}\mspace{14mu} 25}\end{matrix}$

In Equation 25, p_(f,v,D,d)(n) represents the n-th iteration fieldprobability associated with field f, distance d and function D. The summay be over all field values v that appear in any record in field f.Note that Equation 25 may be considered as a weighted sum, where the summay be over all field value probabilities and the weights themselves arealso field value probabilities (hence the squared term). Note furtherthat Equation 25 may be considered as a weighted average of field valueprobabilities. The field probabilities may be converted to field weightsaccording to, by way of non-limiting example:

w _(f,D,d)(n)=−log p _(f,D,d)(n)   Equation 26

Thus, the n-th iteration may calculate field probabilities and fieldweights for every field in every record in the database according toEquations 25 and 26, concluding the second iteration. The fieldprobabilities and field weights may be stored for later use as discussedabove.

Between the n-th iteration and the (n+1)-th iteration, the databaseundergoes a linking operation, which may be based on the match weightsgenerated by the second iteration. The linking operation between then-th iteration and the (n+1)-th iteration may be essentially identicalto the linking operation between the first iteration and the seconditeration. Thus, each record may be compared to every other record inthe database, and each such comparison may result in a link between thecompared records and, therefore, the corresponding pair of entityrepresentations. Each such comparison may result in a link between thecompared pair of records, depending on the results of the comparison.(In some embodiments, every record may be compared with a subset ofother records in the database. Such a subset may be generated usingblocking criteria as disclosed elsewhere herein.) In the exemplarylinking operation, given records r₁ and r₂ and faced with a decision tolink them, Equations 3, 4 or 5 may be used to calculate a score that therecords reference the same individual. If the score exceeds a threshold,the records r₁ and r₂ may be linked. Such a threshold may be determinedas discussed in relation to Equations 3-6. Linking these records links,in turn, the entity representations to which the records may be linked.

It is likely that once the linking operation occurs between the n-thiteration and the (n+1)-th iteration, the number of unique entityrepresentations in the database may be reduced in comparison with thenumber that existed after the n-th iteration.

At block 340, intermediate operations may be performed. Exemplary suchoperations (e.g., transitional linking, propagation, delinking) arediscussed presently.

The database may undergo a transition linking process between the n-thiteration and the (n+1)-th iteration. The transition linking process maybe essentially the same as the transition linking process discussedabove in Section II. The transitional linking process may occur at anystage between iterations, e.g., before or after linking operation 335.

The database may undergo a propagation operation between the n-thiteration and the (n+1)-th iteration. The propagation operation may beessentially the same as the propagation operation that may occur betweenthe first iteration and the second iteration. That is, null field valuesin a first record may be replaced by non-null field values taken from asecond record to which the first record is linked. Likewise, null fieldvalues in the second record may be replaced by non-null field valuestaken from the first record.

The database may undergo a delinking operation between the n-thiteration and the (n+1)-th iteration. Such a delinking operation maydelink records that were incorrectly linked by the linking operationthat preceded the n-th iteration. Exemplary delinking operations aredisclosed in the First Generation Patents And Applications.

Block 345 indicates that one or more of blocks 325, 330, 335 and 340 maybe iterated. The third iteration and subsequent iterations each producefield value probabilities, field value weights, field probabilities, andfield weights. Moreover, the match probabilities and match weightsproduced by each successive iteration may generally be more accuratethan those produced by the prior iteration. The iteration may halt afterany number of iterations after any of blocks 325, 330, 335 or 340. Atblock 350, the match weights and match probabilities may be used to linkrecords as discussed elsewhere herein.

In sum, certain embodiments according to this section produce matchweights that may be used as weights in linking formulas. In computing amatch score for two records according to, for example, Equation 5, thecontents of each field common to the records may be compared. In theevent that the field values of a particular field do not exactly match,that field may be accounted for in the linking formula according to atechnique as disclosed in this section. That is, for such a field (wherethe field values are not identical in the records under comparison), adetermination may be made as to whether the field values are nearmatches as determined by the selected distance function D and forvarious distances d. If the field values do indeed lie within a distanced of each-other according to the distance function D, then the weightused in the matching formula for that particular field may be one ofw_(f,v,D,d)(n) or w_(f,D,d)(n) for some n. Note that here, and ingeneral, the match weight or match probability associated with the leastdistance d may be used. Note that the matching formula may utilizeweights according to a technique disclosed in this section in one ormore terms, and may utilize weights according to other techniquesdisclosed in other sections herein in other terms. Thus, weightsaccording to certain embodiments of this section may be used in amatching formula for a particular field on a case-by-case basis, thatis, depending on whether the field values in the records undercomparison are identical or not.

According to an exemplary embodiment of a technique of this section, amethod of utilizing a record matching formula weight, where the recordmatching formula weight is specific to a particular field value, ispresented. The embodiment includes selecting a symmetric and reflexivefunction and at least one distance. The embodiment calculates a firstprobability that a record in the database includes a field value thatlies within the selected distance of the particular field value asdetermined by the function. The embodiment links records in the databasebased at least in part on the first probability, whereby a plurality ofentity representations are generated. The embodiment calculates a secondprobability that an entity representation in the database includes afield value that lies within the selected distance of the particularfield value as determined by the function. The embodiment links entityrepresentations in the database based at least in part on the secondprobability. And the embodiment allows a user to retrieve informationfrom at least one record in the database.

Various optional features of the above exemplary embodiment include thefollowing. The embodiment may include iterating the calculating a secondprobability and the linking entity representations at least once priorto the retrieving. The embodiment may include calculating a probabilitythat two records match using the record matching formula, where therecord matching formula includes a weighted sum of probabilities thattwo records match, where the weights include the second probability.

According to an exemplary embodiment of a technique of this section, amethod of utilizing a record matching formula weight, the recordmatching formula weight is specific to a particular field andindependent of any particular field value in the particular field, ispresented. The embodiment includes selecting a symmetric and reflexivefunction and at least one distance. The embodiment includes calculatinga plurality of first probabilities, each of the plurality of firstprobabilities reflecting a likelihood that a record in the databaseincludes a field value that lies within the selected distance of adifferent field value as determined by the function. The embodimentincludes calculating a first weight comprising a weighted sum of thefirst probabilities. The embodiment includes linking records in thedatabase based at least in part on the first weight, whereby a pluralityof entity representations are generated. The embodiment includescalculating a second plurality of probabilities, each of the pluralityof second probabilities reflecting a likelihood that an entityrepresentation in the database includes a record comprising a fieldvalue that lies within the selected distance of a different field valueas determined by the function. The embodiment includes calculating asecond weight comprising a weighted sum of the second probabilities. Theembodiment includes linking entity representations in the database basedat least in part on the second weight. The embodiment allows a user toretrieve information from at least one record in the database.

The above described embodiments are exemplary only and are not intendedto limit the scope of the inventions disclosed herein.

IV. Statistical Record Linkage Calibration for Reflexive, Symmetric andTransitive Distance Measures at the Field and Field Value Levels Withoutthe Need for Human Interaction

Embodiments of this technique may be implemented in their own iterativeprocess or incorporated into an iterative process as described above inSection II.

In some embodiments, the techniques of this section provide one or moreweights, which may be used in a record matching formula (e.g., Equations3-5) to scale probabilities (e.g., p_(f) or p_(i)) that two recordscontain nearly matching field values in a particular field. A near matchin field values may be determined at least in part by a reflexive,symmetric and transitive function. In some embodiments, one or moreadditional fields may be added to each record. Each additional field mayserve as a proxy for an original field. A near match in an originalfield may be determined by detecting an exact match in a correspondingproxy field. Moreover, each field value, whether occurring in anoriginal or proxy field, may have associated to it a field valueprobability, which may be converted to a field value weight and used inmaking linking decisions as discussed above in Section II.

Thus, certain embodiments of the present invention associate to each ofone or more select fields a field referred to herein as a “proxy field.”Each proxy field may each contain a field value derived from theoriginal field. A match between field values of two records in a proxyfield may indicate a near match of field values in the original field.Moreover, each field value in each proxy field (and hence eachassociated original field) may have associated to it a field valueprobability, which indicates a probability that a record (respectively,entity representation) chosen at random contains (respectively, containsa record that contains) the same field value in its corresponding proxyfield.

Thus, each proxy field (and hence each associated original field) mayhave associated to it a field value probability, which indicates aprobability that a record (respectively, entity representation) chosenat random contains (respectively, contains a record that contains) asimilar field value in the corresponding original field as determined bythe chosen reflexive, symmetric and transitive distance function. Eachfield value probability may be converted to a field value weightassociated with the relevant proxy field (and hence original field).Such field value weights may be used in making linking decisions asdiscussed above in Section II.

Certain embodiments associate a probability to each field (original andproxy), independent of any particular field value. For a given field,the associated probability may be computed as a weighted average of theprobabilities associated with each individual proxy field value that mayoccur in the proxy field. (Such probabilities may also be associatedwith the field values appearing in the original field.) Moreover, thefield probabilities calculated by certain embodiments may be convertedto field weights and used in making record linking decisions. Suchdecisions may take into account some or all of the fields common to therecords. In this technique, knowledge of the common field values may benot required. Further, this technique produces accurate results for anytwo records, regardless as to the contents of their fields.

The present technique may use various measures of similarity. That is,the present technique is not limited to a single measure of near matchesbetween field values. Instead, any reflexive, symmetric and transitivefunction may be used to detect or measure similarity of field values. Anexample of such a function is SOUNDEX. The SOUNDEX function takes astring as an argument and outputs a code in standard format thatprovides an indication of the string's pronunciation. The output of theSOUNDEX function (or any other reflexive, symmetric and transitivefunction) may be referred to herein as a “code.” Note that, in general,reflexive, symmetric and transitive functions define a partition of thedomain over which the function operates, where the partition may bedefined according to the codes assigned to elements of the domain by thefunction. That is, each part of the partition may be defined by adifferent code assigned only to the elements in that part by thefunction. The SOUNDEX function is reflexive because it produces the samecode every time the same string is input. It is symmetric because if twostrings produce the same code, they will produce the same coderegardless as to the order of computation, i.e., regardless as to whichstring is fed into the SOUNDEX function first. The SOUNDEX function istransitive because if a first string and a second string produce thesame code, and if the second string and a third string produce the samecode, then the first string and the third string produce the same code.Note that the edit distance function is not transitive. For example, theedit distance between the strings “tape” and “tale” is one, and the editdistance between the strings “tale” and “tall” is one, but the editdistance between the string “tape” and “tall” is two, rather than one.For the remainder of this section, the term D will denote a functionwith the appropriate properties, not limited to SOUNDEX. Note that unaryfunctions or binary functions may be used with the present technique.

Certain embodiments of the present technique add one or more fields toeach record in order to associate codes with each of one or moreexisting field values in the record. Thus, by way of non-limitingexample, each record may be appended with an additional field thatcontains as its field value the SOUNDEX code for that record's FirstName field value. Continuing the example, each record may further beappended with another field that contains as its field value the SOUNDEXcode of that record's Last Name field value. An arbitrary number ofadditional fields may be appended to the records in this manner. Moreparticularly, the technique of this section may be applied to any fieldthat is amenable to a near-match or similar field value analysis. By wayof non-limiting example, for each record, the field value of any suchfield may be input in the selected distance function D, and the outputmay be included in an additional field appended to such record. Again,the technique is not limited to SOUNDEX. Rather, a code produced by anyfunction D with the appropriate properties may be included in suchadditional field(s).

Thus, for each record, certain embodiments derive a code from the fieldvalue of an existing field and insert such code into a proxy field. Suchembodiments may repeat this process for a plurality of fields.Accordingly, certain embodiments add one of more fields to each record.An iteration, such as that discussed above in Section II, may then beperformed on the altered records in order to compute match probabilitiesand match weights for the field values in the proxy fields. The fieldvalues in such added fields may be used in making linking decisions.

FIG. 4 is a flowchart depicting an exemplary embodiment of an inventionof Section IV. The present embodiment may be implemented in conjunctionwith an embodiment of the techniques of Section II. For purposes ofillustration rather than limitation, the present embodiment will bediscussed in reference to records r₁ and r₂ reflected in the tablebelow:

Record First Name Last Name SSN r₁ John Smiff 999-99-9999 r₂ Jon Smith999-99-9999

At block 405, a transitive, symmetric and reflexive function isselected. For purposes of illustration rather than limitation, theSOUNDEX function will be used in the present embodiment.

At block 410, one or more fields are selected, and corresponding proxyfield(s) are created and populated. Thus, one or more fields in whichnear matches will be analyzed are chosen. Examples of such fieldsinclude First Name, Last Name, Social Security Number, and others. Forpurposes of illustration rather than limitation, the present embodimentwill be discussed in reference to the First Name and Last Name fields asthe selected fields, with the understanding that the invention is notlimited to such fields.

Once the fields are selected, the exemplary embodiment adds a proxyfield to each record for each selected field (block 410). In the exampleunder discussion, the records may appear as:

 Last Proxy First Proxy Last Record First Name Name SSN Name Name r₁John Smiff 999-99-9999 r₂ Jon Smith 999-99-9999

The exemplary embodiment proceeds by determining codes for each fieldvalue in each selected field and inserting such codes into the proxyfields. Now, the SOUNDEX code for “John” may be J500, the SOUNDEX codefor “Smiff” may be S510, the SOUNDEX code for “Jon” may be J500, and theSOUNDEX code for “Smith” may be S530. Thus, the embodiment proceeds byentering such codes as field values in the appropriate proxy fields asfollows:

Last Proxy First Proxy Last Record First Name Name SSN Name Name r₁ JohnSmiff 999-99-9999 J500 S510 r₂ Jon Smith 999-99-9999 J500 S530

At block 415, match probabilities and match weights are calculated.Thus, once the codes are entered in the associated proxy fields, theembodiment may proceed by implementing a technique of Section II todetermine match probabilities and match weights for each field or fieldvalue. That is, once proxy fields and the appropriate field values areadded to the records, the embodiment may proceed as discussed above inSection II in order to determine one or more of field valueprobabilities, field probabilities, match value weights and matchweights. That is, an iteration such as that discussed in Section II maybe performed on the altered records in order to compute matchprobabilities and match weights. For computing match weights and matchprobabilities, the iteration essentially treats the proxy fields andtheir included field values the same as if such fields and field valueswere originally in the records instead of having been added. Theiteration may include one or more of the steps set forth in Section II,such as calculating field value probabilities and field value weights(based on a number of records), calculating field probabilities andfield weights (based on a number of records), preliminary linkingoperations, initial intermediate operations, calculating field valueprobabilities and field value weights (based on a number of entityrepresentations), calculating field probabilities and field weights(based on a number of entity representations), linking operations andintermediate operations. The computed match weights may be used inmaking linking decisions as discussed in Section I.

The field value probabilities and field value weights may be stored forlater use. For example, these parameters may be stored in a lookuptable, alone or together with other relevant parameters, such as theproxy field values. Alternately, or in addition, these parameters may bestored in one or more fields added to each record. By way ofnon-limiting example, field value weights may be stored in fields addedto records in which the associated field values appear. The parametersmay be updated with each iteration by replacing parameters from prioriterations or by adding newly generated parameters. In some embodiments,one or both of field value probabilities and field value weights may bestored in fields appended to records, while one or both of fieldprobabilities and field weights may be stored in one or more lookuptables.

At block 420, the calculated match weights and match probabilities maybe used to link records as discussed elsewhere herein. For example, theproxy fields may be accounted for in a matching formula as follows.Equations 3-5 provide match scores for two records as weighted sums. Asdiscussed in Section I, if a match score exceeds a threshold, therecords under consideration may be linked. The weighted sums in thematching formulas may generally weight probabilities of field valuematches by field value or field weights associated with the field valueor field, respectively. This process may be used for the proxy fields asdisclosed in this section. That is, the proxy fields may be treated asany other field in determining a match between records.

Alternately, the proxy fields may be accounted for in a matching formulaas follows. In comparing two records, the field values in the originalfields may be compared prior to comparing the field values in the proxyfields. If the field values in an original field are identical betweenthe records, then the proxy field values may not be compared, and a termfor the proxy field may be omitted from the matching formula. That is,the matching formula may include a term p_(i)w_(i) corresponding to theoriginal field, and omit a term p_(j)w_(j) that corresponds to the proxyfield. On the other hand, in comparing the two records, if the fieldvalues in an original field are not identical, then the proxy fieldvalues may be compared. If the proxy field values match, then a termw_(j)p_(j) for the proxy field may be included in the matching formulain place of the term for the original field. If the field values in theproxy field match, the associated probability p_(j) may be set to one(1) and the weight w_(j) may be a field weight or field value weightcorresponding to the proxy field or the field value therein,respectively. Alternate techniques for setting the value of p_(j) arefound in the First Generation Patents And Applications.

Another alternate technique for accounting for proxy fields in amatching formula is discussed presently. As above, this discussion isrelative to two records for which a linking decision is to be made.Assume for purposes of illustration that the original field has index iand that the associated proxy field has index j. Then, instead ofincluding one or both terms p_(i)w_(i) and p_(j)w_(j) in the matchingformula, the following term may be used instead:p_(i)w_(i)+p_(j)(1−p_(i))w_(j). Note that this term is equal tow_(i)p_(i) whenever p_(i) equals one (e.g., whenever field values in theoriginal field match). Note further that this term is equal tow_(j)p_(j) whenever p_(i) equals zero (e.g., if field values in theoriginal field do not match and the embodiment in which the matchingformula is implemented sets p_(i) equal to zero in such instances). Inembodiments where one or both terms p_(i) and p_(j) are allowed to havevalues between zero and one (e.g., as set forth in the First GenerationPatents And Applications) the term essentially blends w_(i)p_(i) with aportion of w_(j)p_(j).

According to an exemplary embodiment of a technique of this section, amethod of utilizing a record matching formula weight, where the recordmatching formula weight is specific to a particular field valueassociated with a particular field, the method for use with a databasecomprising a plurality of records, is presented. The embodiment includesselecting a symmetric, reflexive and transitive function, wherebyapplying the function to field values appearing in the particular fieldin the plurality of records defines a first partition of the pluralityof records, where the first partition includes a first plurality offirst parts, each of the first parts being associated with at least onefield value appearing in the particular field. The embodiment includescalculating a first probability that a record in the database is in afirst part associated with the particular field value. The embodimentincludes linking records in the database based at least in part on thefirst probability, whereby a plurality of entity representations aregenerated, whereby applying the function to field values appearing inthe particular field in the plurality of entity representations definesa second partition of the plurality of entity representations, where thesecond partition includes a second plurality of second parts, each ofthe second parts associated with at least one field value appearing inthe particular field. The embodiment includes calculating a secondprobability that an entity representation in the database is in a secondpart associated with the particular field value. The embodiment includeslinking entity representations in the database based at least in part onthe second probability. The embodiment includes allowing a user toretrieve information from at least one record in the database.

According to an exemplary embodiment of a technique of this section, amethod of utilizing a record matching formula weight, where the recordmatching formula weight is specific to a particular field andindependent of any particular field value in the particular field, themethod for use with a database comprising a plurality of records, ispresented. The embodiment includes selecting a symmetric, reflexive andtransitive function, whereby applying the function to field valuesappearing in the particular field in the plurality of records defines afirst partition of the plurality of records, where the first partitionincludes a first plurality of first parts, where each of the first partsis associated with at least one field value appearing in the particularfield. The embodiment includes calculating a plurality of firstprobabilities, each of the plurality of first probabilities reflecting alikelihood that a record in the database is in a different first part.The embodiment includes calculating a first weight comprising a weightedsum of the first probabilities and linking records in the database basedat least in part on the first weight, whereby a plurality of entityrepresentations are generated, whereby applying the function to fieldvalues appearing in the particular field in the plurality of entityrepresentations defines a second partition of the plurality of entityrepresentations, where the second partition includes a second pluralityof second parts, each of the second parts associated with at least onefield value appearing in the particular field. The embodiment includescalculating a second plurality of probabilities, each of the pluralityof first probabilities reflecting a likelihood that a record in thedatabase is in a different first part. The embodiment includescalculating a second weight comprising a weighted sum of the secondprobabilities. The embodiment includes linking entity representations inthe database based at least in part on the second weight. The embodimentincludes allowing a user to retrieve information from at least onerecord in the database.

The above described embodiments are exemplary only and are not intendedto limit the scope of the inventions disclosed herein.

V. Statistical Record Linkage Calibration for Interdependent FieldsWithout the Need for Human Interaction

Embodiments of this technique may be implemented in their own iterativeprocess or incorporated into an iterative process as described above inSection II.

Some embodiments account for the phenomenon of interdependent fields.For example, certain field values are, at least to some extent,correlated with other field values. The correlation may be positive ornegative. Thus, certain field values may tend to imply or suppress otherfield values. Thus, for example, a Gender field value of Male is likelyto have a weak positive correlation with a First Name field value of“John”; whereas the same field value is likely to have a weak negativecorrelation with a First Name field value of “Mary”. As another example,a City field value of “Boca Raton” may have essentially the samesignificance as a City field value of “Boca Raton” coupled with a Statefield value of “Florida,” whereas a City field value of “Jacksonville”alone may be much less significant (e.g., by a factor of ten) incomparison with a City field value of “Jacksonville” coupled with aState field value of “Florida.” Thus, certain fields (and field values)may be interdependent, and certain embodiments of the present techniqueaccount for such interdependence. Such embodiments may generally producesuperior results in comparison with techniques that assume that fieldsare independent.

Certain embodiments of the present invention provide a separate,individual statistical significance to a combination of fields. Thecombination may be fields that are statistically correlated oranti-correlated.

In some embodiments, the techniques of this section provide one or moreweights, which may be used in a record matching formula (e.g., Equations3-5) to scale probabilities (e.g., p_(f) or p_(i)) that two recordscontain a matching combination of particular field values in a pluralityof fields.

In some embodiments, one or more supplemental fields may be added toeach record. Each such supplemental field may account for the contentsof a plurality of other fields. Each supplemental field allows certainembodiments of the present invention to accord a single statisticalsignificance to a combination of fields. Moreover, each field value,whether occurring in an original or supplemental field, may haveassociated to it a field value probability, which may be converted to afield value weight and used in making linking decisions as discussedabove in Section II. That is, an iteration, such as that discussed abovein Section II, may be performed on the altered records in order tocompute match probabilities and match weights for the field values inthe supplemental fields. Thus, the significance of an improbablecombination of field values may be determined to be high, whereas thesignificance of a probable combination of field values may be determinedto be low.

Note that more than one such supplemental field may be added. Thus, byway of non-limiting example, each record may be appended with anadditional field that contains as its field value an amalgamation ofFirst Name, Middle Name and Last Name field values. Continuing theexample, each record may further be appended with another field thatcontains as its field value an amalgamation of Last Name, City andStreet Address field values. An arbitrary number of additional fieldsmay be appended to the records in this manner. The amalgamation may beaccomplished using any of a variety of techniques, such as, by way ofnon-limiting example, concatenation, linked list, use of a hashfunction, use of separator characters, etc.

Thus, certain embodiments of the present invention associate to each ofa plurality of select fields a supplemental field. Each supplementalfield may each contain a field value derived from a plurality of fieldvalues. Moreover, each supplemental field may have associated to it afield value probability, which indicates a probability that a record(respectively, entity representation) chosen at random contains(respectively, contains a record that contains) the associated fieldvalue in the associated field. Each field value probability may beconverted to a field value weight associated with the relevantsupplemental field. Such field value weights may be used in makinglinking decisions as discussed above in Section II.

Certain embodiments associate a probability to each supplemental field,independent of any particular field value. For a given supplementalfield, the associated field probability may be computed as a weightedaverage of the probabilities associated with each individual field valuethat may occur in the given supplemental field. Moreover, the fieldprobabilities calculated by certain embodiments may be converted tofield weights and used in making record linking decisions. Suchdecisions may take into account some or all of the fields common to therecords. In this technique, knowledge of the common field values may notbe required. Further, this technique produces accurate results for anytwo records, regardless as to the contents of their fields.

The field value probabilities and field value weights may be stored forlater use. For example, these parameters may be stored in a lookuptable, alone or together with other relevant parameters. Alternately, orin addition, these parameters may be stored in one or more fields addedto each record. By way of non-limiting example, field value weights maybe stored in fields added to records in which the associated fieldvalues appear. The parameters may be updated with each iteration byreplacing parameters from prior iterations or by adding newly generatedparameters. In some embodiments, one or both of field valueprobabilities and field value weights may be stored in fields appendedto records, while one or both of field probabilities and field weightsmay be stored in one or more lookup tables.

The supplemental fields may be accounted for in a matching formula asfollows. Equations 3-5 provide match scores for two records as weightedsums. As discussed in Section I, if a match score exceeds a threshold(e.g., as computed using Equation 6), the records under considerationmay be linked. The weighted sums in the matching formulas may generallyweight probabilities of field value matches by field value or fieldweights associated with the field value or field, respectively. Thisprocess may be used for the supplemental fields as disclosed in thissection. That is, the supplemental fields may be treated as any otherfield in determining a match between records.

Alternately, the supplemental fields may be accounted for in a matchingformula as follows. In comparing two records, the field values in thesupplemental fields may be compared prior to comparing the field valuesin the original fields. If, on the one hand, the field values in asupplemental field are identical between the records, then a term forthe supplemental field may replace the terms for the constituent fieldsin the matching formula. That is, the matching formula may include aterm p_(i)w_(i) corresponding to the supplemental field, and omit termsp_(k)w_(k), . . . , p_(l)w_(l) that correspond to the fields that makeup the supplemental fields. If the field values in the supplementalfield match, the term pi may be set to one (1) and the weight w_(i) maybe a field weight or field value weight corresponding to thesupplemental field or the field value therein, respectively. If, on theother hand, in comparing the two records the field values in asupplemental field are not identical, then a term for the supplementalfield may be omitted from the matching formula, and terms for theconstituent field may be included instead. In that instance, the termsfor the constituent fields (referred to here as p_(k)w_(k), . . . ,p_(l)w_(l)) may each be scaled by multiplication by a supplementalweight as follows. Such a supplemental weight may be computed as a matchweight for the supplemental field divided by the sum of match weightsfor the constituent fields. Such match weights may be field weights orfield value weights. Thus, the supplemental weight may be computed as,by way of non-limiting example:

$\begin{matrix}{W = {\frac{w_{i}}{w_{k} + \ldots + w_{l}}.}} & {{Equation}\mspace{14mu} 27}\end{matrix}$

In Equation 27, the term w_(i) represents a field value weight for thefield value in the supplemental field, and the terms w_(k), . . . ,w_(l) represent the field value weights for the field values in theconstituent fields. (Alternately, the term w_(i) may represent a fieldweight for the supplemental field, and the terms w_(k), . . . , w_(l)may represent the field weights for the constituent fields.) Note that Was set forth above may be a measure of interdependence of theconstituent field values (respectively, fields). That is, if thesignificance of the supplemental field value (respectively, field)exceeds the sums of the significances of the individual constituentfield values (respectively, fields), then W will be greater than one.This situation may be expected to happen for the example provided aboveof a combination of field values Gender=Male and First Name=Mary.Otherwise, W may be less than one, indicating that the constituent fieldvalues may be at least weakly correlated. The matching formula terms forthe constituent fields may be modified in the instance where there isnot an exact match in supplemental field values to appear asWp_(k)w_(k), . . . , Wp_(l)w_(l), where the terms may be as definedabove in relation to Equation 27. That is, if there is no match in asupplemental field, the matching formula term for the supplemental fieldmay be omitted in favor of terms for the constituent fields weighted bya term as set forth above in Equation 27, which may be a measure ofcorrelation between the constituent field values or fields. Note that Wmay be specific to each record, and each W may be stored in one or bothof a lookup table and in a field appended to the associated record.

An alternate technique for accounting for supplemental fields in amatching formula is discussed presently. As above, this discussion isrelative to two records for which a linking decision is to be made.Assume for purposes of illustration that the supplemental field hasindex i and that the associated constituent fields has indexes j, . . ., l. Then instead of including some or all of terms p_(i)w_(i),p_(k)w_(k), . . . , p_(l)w_(l) in the matching formula, the followingterm may be substituted: p_(i)w_(i)+(1−p_(i))(Wp_(k)w_(k)+ . . .+Wp_(l)w_(l)). Note that this term is equal to w_(i)p_(i) whenever p_(i)equals one (e.g., whenever field values in the original field match).Note further that this term is equal to Wp_(k)w_(k)+ . . . +Wp_(l)w_(l)whenever p_(i) equals zero (e.g., if field values in the supplementalfield do not match and the embodiment in which the matching formula isimplemented sets p_(i) equal to zero in such instances). In embodimentswhere some or all terms p_(i), p_(k), . . . , p_(l) may be allowed tohave values between zero and one (e.g., as set forth in the FirstGeneration Patents And Applications) the term essentially blendsw_(i)p_(i) with a portion of Wp_(k)w_(k)+ . . . +Wp_(l)w_(l).

In some embodiments, in the event of a non-match in a supplementalfield, techniques that handle near matches, for example, the techniquesset forth in Sections III, IV or X, may be applied to the supplementalfield.

FIG. 5 is a flowchart depicting an exemplary embodiment of an inventionof Section V. In general, embodiments according to this section may beimplemented in conjunction with an embodiment of the techniques ofSection II.

For purposes of illustration rather than limitation, the presentembodiment will be discussed in reference to the record reflected in thetable below:

First Middle Name Name Last Name SSN Street Address City John Sue Smith999-99-9999 321 Fake Street Anytown

At block 505, the exemplary embodiment commences by choosing a pluralityof fields to amalgamate into a supplemental field. In this embodiment,this step may be done twice, however, this step may be performed anynumber of times, not limited to two.

At block 510, one or more corresponding supplemental fields are createdand populated. For purposes of illustration rather than limitation, afirst supplemental field will be added with the First Name, Middle Nameand Last Name fields as the selected fields, and second supplement fieldwill be added with the Last Name and Street Address fields as theselected fields. The resulting record with the two added supplementalfields may appear as follows:

First Middle Last Street Supplemental Supplemental Name Name Name SSNAddress City Field 1 Field 2 Jon Sue Smith 999- 321 AnytownJon/Sue/Smith Smith/321FakeStreet 99- Fake 9999 Street

At block 515, match probabilities and match weights are calculated. Theembodiment may proceed by implementing an iterative technique of SectionII to determine match probabilities and match weights for each field orfield value. That is, once supplemental fields and the appropriate fieldvalues are added to the records, the embodiment may proceed with aniteration as discussed above in Section II in order to determine one ormore of field value probabilities, field probabilities, match valueweights and match weights. For computing match weights and matchprobabilities, the iteration essentially treats the supplemental fieldsand their included field values the same as if such fields and fieldvalues were originally in the records instead of having been added. Theiteration may include one or more of the steps set forth in Section II,such as calculating field value probabilities and field value weights(based on a number of records), calculating field probabilities andfield weights (based on a number of records), preliminary linkingoperations, initial intermediate operations, calculating field valueprobabilities and field value weights (based on a number of entityrepresentations), calculating field probabilities and field weights(based on a number of entity representations), linking operations andintermediate operations. The weights computed by the iteration may beused in making linking decisions as discussed in Section I.

More particularly, an iteration that includes a technique of thissection may proceed as follows. The first iteration may take place onceone or more additional fields are added to each record and populatedwith the appropriate field values as discussed above. After eachiteration, a propagation process may occur as discussed in Section II,for example. After such propagation process, the field values of the oneor more supplemental fields and their associated match weights may beupdated. This supplemental field updating after a propagation processserves to ensure that the supplemental fields contain information thathas been propagated. The propagation and supplemental field updating mayoccur after each iteration.

At block 520, the calculated match weights and match probabilities maybe used to link records as discussed elsewhere herein or as discussedpresently. For example, between each iteration, the linking process maybe proceed as follows. Note that such linking process may utilize atechnique of comparing field values between two records as discussed indetail above in Section I (e.g., in reference to Equations 3-6). Thecomparison of two records that have been modified as illustrated in thenon-limiting example above by adding two supplemental fields mayinitially compare field values in the supplemental fields. If the fieldvalues in the supplemental fields of two records exactly match, then thecomparison may omit comparing the individual field values that may beaccounted for in the supplemental fields.

By way of non-limiting example, consider a comparison (for the purposeof determining whether to link records as discussed above in Section I)of the exemplary above record with another record that has also beenmodified with the addition of two supplemental fields. Such a comparisonmay proceed by comparing the field values of the supplemental fieldsprior to comparing the contents of the fields that make up the contentsof the supplemental fields. Suppose that, in this exemplary comparison,there is an exact match in Supplemental Field 1, but not in SupplementalField 2. In such an instance, the comparison may proceed by accountingfor the match in Supplemental Field 1 in, for example, Equation 5. Moreparticularly, because the contents of Supplemental Field 1 match, theassociated probability p_(i) may be set equal to one, and the fieldvalue weight w_(i) associated with the Supplemental Field 1 field valuemay be utilized in the weighted sum of Equation 5, where the subscript iin this sentence is the index for Supplemental Field 1. (The fieldweight associated with Supplemental Field 1 may be used in thealternative.) That is, the term p_(i)w_(i) for the supplemental fieldmay be included in a matching formula, and the terms for the constituentfields may be omitted. With Supplemental Field 1 already accounting forthe First Name, Middle Name and Last Name fields, terms for these fieldsmay be omitted from the weighted sum of the matching formula. Thus, theweighted sum of, for example, Equation 5, may omit the indexes for theFirst Name, Middle Name and Last Name fields from the set of indexesover which to sum, as the field values in these fields have already beenaccounted for in Supplemental Field 1.

Turning now to Supplemental Field 2, because there is no exact match inSupplemental Field 2 in this example, a term for this field may beomitted from the weighted sum of the relevant matching formula (forexample, Equation 5). Note that Supplemental Field 2 includes fieldvalues from the Last Name field and the Street Address field. Asdiscussed above, in this example, while the Last Name field is accountedfor in the matching formula by the term corresponding to SupplementalField 1, the Street Address field is not. However, the Street Addressfield may be accounted for in a matching formula separately. That is, aterm for the Street Address field may be included in, for example,Equation 5. The Street Address term may include a product of aprobability p_(j) of a match and a match weight w_(j), where j is anindex corresponding to the Street Address field. The match weight w_(j)may be a field value weight or a field weight. The Street Address termmay further include an additional weight W, which adjusts for the amountof interdependence among field values (respectively, fields).

In short, because the field values in Supplemental Field 1 are assumedto be identical in this example, a matching formula may omit terms forthe constituent fields of Supplemental Field 1, while including a termfor Supplemental Field 1 itself. The term for Supplemental Field 1 mayinclude a probability p_(i) of a match (which may be set equal to onebecause of the assumed exact match in this example) multiplied by thefield or field value weight for Supplemental Field 1. Because the fieldvalues in Supplemental Field 2 are assumed not to be identical in thisexample, a matching formula may omit a term for Supplemental Field 2,while including terms for the constituent fields that have not otherwisebeen accounted for (e.g., in the term for Supplemental Field 1),weighted by a term that adjusts for the amount of correlation among theconstituent fields.

According to an exemplary embodiment of a technique of this section, amethod of utilizing a record matching formula weight, where the recordmatching formula weight is specific to a particular plurality of fieldvalues, is presented. The embodiment includes selecting a plurality offields and adding a supplemental field to each of a plurality ofrecords. The method includes populating the supplemental field, for eachof the plurality of records, with a field value representative of fieldvalues from each of the plurality of fields in the record, whereby thesupplemental field of at least one record contains a particular fieldvalue representative of the particular plurality of field values. Theembodiment includes calculating a first probability that a record in thedatabase includes the particular field value in a supplemental field ofthe record. The embodiment includes linking records in the databasebased at least in part on the first probability, whereby a plurality ofentity representations are generated. The embodiment includescalculating a second probability that an entity representation in thedatabase includes the particular field value in a supplemental field ofa record linked to the entity representation. The embodiment includeslinking entity representations in the database based at least in part onthe second probability. The embodiment includes allowing a user toretrieve information from at least one record in the database.

According to an exemplary embodiment of a technique of this section, amethod of utilizing a record matching formula weight, where the recordmatching formula weight is specific to a plurality of fields andindependent of any particular field value in the particular plurality offields, is presented. The embodiment includes selecting a plurality offields and adding a supplemental field to each of a plurality ofrecords. The embodiment includes populating the supplemental field, foreach of the plurality of records, with a field value representative offield values from each of the plurality of fields in the record. Theembodiment includes calculating a plurality of first probabilities, eachof the plurality of first probabilities reflecting a likelihood that arecord in the database includes a particular field value in asupplemental field. The embodiment includes calculating a first weightcomprising a weighted sum of the first probabilities. The embodimentincludes linking records in the database based at least in part on thefirst weight, whereby a plurality of entity representations aregenerated. The embodiment includes calculating a second plurality ofprobabilities, each of the second plurality of probabilities reflectinga likelihood that an entity representation in the database includes aparticular field value in a supplemental field. The embodiment includescalculating a second weight comprising a weighted sum of the secondprobabilities. The embodiment includes linking entity representations inthe database based at least in part on the second weight. The embodimentincludes allowing a user to retrieve information from at least onerecord in the database.

The above described embodiments are exemplary only and are not intendedto limit the scope of the inventions disclosed herein.

VI. Automated Detection of Null Field Values and Effectively Null FieldValues

In some embodiments, the technique of this section provides a numericalcritical frequency associated with a field that may be used to detectfield values that may be treated as null. That is, for a field, certainembodiments provide a critical frequency such that a field valueassociated with the field that occurs more than (or, in someembodiments, equal to) the critical frequency may be treated as a nullfield value, and a field value associated with the field that occursless than (or, in other embodiments equal to) the critical frequency maybe treated as a non-null field value. In such embodiments, a separatecritical frequency may be calculated for each field. For example, acritical frequency may be calculated for and/or associated with a LastName field, while a separate critical frequency may be calculated forand/or associated with a Gender field.

Note that embodiments according to this section may be incorporated intoany of the embodiments described in any section herein.

FIG. 6A is a flowchart depicting an exemplary embodiment according tothis section. At block 605, fields to which the present technique are tobe applied are selected.

At block 610, the number of different field values present in records inthe database are counted for each selected field. In general, thetechnique of calculating a critical frequency for one or more fields maybegin by counting, for each such field, the number of records thatinclude each field value. That is, for every field value that appears ina given field in any record in the database, the technique ofcalculating a critical frequency may begin by determining the number ofrecords in the database that include each field value in the associatedfield. These counts may be used to form a separate histogram for eachfield. As described further below, such histograms may then be used todetermine the critical frequency.

At block 615, a critical frequency is calculated for each selectedfield. In general, a critical frequency may be calculated using datagenerated from a pre-processing step (e.g., pre-linking step, etc.) thatmay be independent of the exemplary embodiment of the inventiondescribed in Section I. In some embodiments, a critical frequency may becalculated using data generated from a processing step (e.g., linkingstep, etc.) that may be associated with an embodiment of the inventiondescribed in Section II. In such embodiments, the technique ofcalculating a critical frequency for each field of a database may beginby accessing the counts determined in the first iteration of thetechnique described in Section II. These counts may be used to form aseparate histogram for each field. As described further below, suchhistograms may then be used to determine the critical frequency.

FIG. 6B is an exemplary histogram for a Last Name field. Note that thex-axis corresponds to the various different last names present in theLast Name field in any record in the database, whereas the y-axiscorresponds to the count of such last names. Note further that the fieldvalue counts may be arranged in decreasing order. Thus, the field valueshaving the highest frequencies appear toward the left, while the fieldvalues having the smallest frequencies appear toward the right. Forexample, for the Last Name field, the “blank” field value (denoted inthe chart above as “[ ]”), “N/A” and “not applicable” have the highestfrequency counts and therefore appear toward the left. Conversely, forthe Last Name field, uncommon field values such as “Broflovski” may beassociated with the smallest frequency values, and therefore may appeartoward the right-hand side.

In some embodiments, the technique of calculating a critical frequencyfor each field of a database may continue by calculating the differencebetween adjacent frequencies. More particularly, if the function definedby the above histogram is denoted as g, then the difference in valuebetween adjacent frequency values may be represented as, by way ofnon-limiting example:

f(x)=g(x)−g(x+1).   Equation 28

In the above frequency value difference formula, f represents a functionof the frequency value differences calculated using the function gdefined by the histogram, where g(v) represents a frequency valueassociated with the v-th field value. For example, f may be calculatedusing Equation 28 based on the exemplary Last Name field data displayedin the above histogram. More particularly, f may be calculated as, byway of non-limiting example:

v 1 2 3 4 5 6 7 8 9 10 11 12 13 14 g(v) 14 11 8.5 8.0 6.0 4.0 3.7 3.63.5 3.0 2.0 2.0 2.0 1.8 g(v + 1) 11 8.5 8.0 6.0 4.0 3.7 3.6 3.5 3.0 2.02.0 2.0 1.8 1.7 f(v) 3.0 2.5 0.5 2.0 2.0 0.30 0.10 0.10 0.50 1.0 0.000.00 0.20 0.10

FIG. 6C is an exemplary graph of f. The critical frequency above whichfield values may be considered as null may be calculated using f. Insome embodiments, the critical frequency may be the first point at whichthe derivative (e.g., calculus derivative) of f changes from negative topositive. More generally, the critical frequency may be the point atwhich the derivative of f changes from a first sign (e.g., negative,positive) to a second sign (e.g., positive, negative), where the secondsign is different from the first sign. As is known in the art, the pointat which the derivative of f changes from a first sign to a second signmay be determined by observing the point at which f changes from anincreasing function to a decreasing function or the point at which fchanges from a decreasing function to an increasing function. By way ofnon-limiting example, the point at which f changes from a decreasingfunction to an increasing function is illustrated in FIG. 6D.

In FIG. 6D, the dotted line depicts the point at which the f firstchanges from a decreasing function to an increasing function(accordingly, the point at which the derivative of f changes sign fromnegative to positive). Any field value whose frequency is greater thanor equal to the frequency corresponding to that point may be consideredas a null field value. Any field value occurring less often than thecritical frequency may be treated as a non-null field value. Note that,as revealed by an inspection of the graph of g far above, the criticalfrequency associated with the Last Name field values is approximately8,200. Accordingly, for this embodiment, all field values appearing8,200 times or more may be treated as null field values. All fieldvalues appearing less than 8,200 times in any record in a database maybe treated as non-null field values.

In some embodiments, the critical frequency may be determined to be thepoint at which the derivative of f first equals zero. In someembodiments, the critical frequency may be determined to be the point atwhich the derivative of f decreases below a threshold. Such a thresholdmay be, by way of non-limiting example, 10, 20, 50, 100, 200, 500, 1000,2000, 5000 or 10,000. In some embodiments, the critical frequency may bedetermined by transforming f into a continuous function using, e.g., aleast-squares approach, and then calculating the derivative of thecontinuous function and detecting where it changes signs as explainedabove.

At block 620, field values that appear more than the critical frequencyare considered to be null. Field values that have been determined to benull or equivalent to null may be replaced by a special character, acanonical null value, deleted from the field, or accounted for usinganother technique such as recordation of each instance of such a valuein a lookup table. In some embodiments, the field values are leftunchanged, but are treated as null in any technique presented hereinthat distinguishes between null values and non-null values.

The above described embodiments are exemplary only and are not intendedto limit the scope of the inventions disclosed herein.

VII. Adaptive Clustering of Records and Entity Representations

Embodiments of this technique may be implemented in their own iterativeprocess, incorporated into a non-iterative process, or incorporated intoan iterative process such as described above in Section II.

FIG. 7 is a flow diagram depicting an exemplary technique foridentifying and linking related records in accordance with at least oneembodiment of the invention of this section. The technique of FIG. 7 maybe used as a transition process, which may be implemented at, forexample, step 412 of FIG. 4 in the First Generation Patents AndApplications or any of blocks 130, 215, 220, 235, 240, 250, 315, 320,335, 340, 350, 420 and 520 of the present disclosure. More generally,the technique of FIG. 7 may be utilized during a link phase to identifyindirect links between records. Once identified, such indirect links maybe implemented by linking together the identified records (e.g., linkinga record to an entity representation) as described elsewhere herein. Ingeneral, an embodiment of the invention of this section may beimplemented as a transitional linking process or a record linkingprocess in any of the iterative techniques presented in this document.The techniques of this section may be used to pick the best records tolink from among a pool of records generated by any of the techniquesdisclosed or incorporated by reference herein. The technique of FIG. 7may be implemented in addition to, or instead of, one or more of thetechniques described above in reference to FIGS. 8-10 in the FirstGeneration Patents And Applications.

The technique of FIG. 7 may be applied as part of an iterative process,for example, a process as described in Section II. By way ofnon-limiting example, a first iteration in such a process may includeprocessing each record in the database, as at this stage, the recordsmay not be linked at all. Thus, for the first iteration, each record maybe compared with every other record for the purpose of calculating amatch score for every pair of records and detecting related records.Subsequent iterations may only calculate match scores for, and link,pairs of records that are themselves linked to different entityrepresentations. That is, iterations after the first iteration may onlycompare pairs of records where each record is linked to a differententity representation (or where at least one record is unlinked).Accordingly, the techniques of this section may be applied to all, orless than all, of the records present in the database. Thus, for a firstiteration, every pair of records taken from the entire universe ofrecords may be processed. In subsequent iterations, only a subset ofpairs of records may be assigned match scores. By way of non-limitingexample, in some embodiments, subsequent iterations may process eachentity representation separately. That is, for a given entityrepresentation, only pairs of records that include at least one recordalready linked to the given entity representation may be processed.

Table VII.1 below illustrates an exemplary database prior to any linkingof records, where only a selected subset of data fields is represented.A first exemplary iteration is discussed presently. The term “DID” means“definitive identifier” as that term finds meaning in the FirstGeneration Patents And Applications, however, embodiments of the presentinvention are not limited to utilizing DIDs for linking or identifyingrecords or entity representations. The term “RID” means “entityreference identifier” as that term finds meaning in the First GenerationPatents And Applications; however, the term “RID” may alternately mean“record identifier,” an identifier associated with each record. In thisexample, the data fields include the first name (Fname) data field, thelast name (Lname) data field, the date-of-birth (DOB) data field, thestreet address (Stad) data field and the SSN data field.

TABLE VII.1 Row No. DID RID Fname Lname DOB Stad SSN 1 1 1 Mary James 7Main St. 123456789 2 2 2 Mary James 19970606 7 Main St. 3 3 3 Mary James19670923 7 Main St. 4 4 4 Mary James 19970606 7 Main St. 987654321 5 5 5Mary James 19670923 7 Main St. 123456789 6 6 6 Mary James 7 Main St.

At block 705, pairs of records may be assigned a match score. The matchscores may be stored in a match table, an example of which is presentedbelow as Table VII.2. For simplicity of discussion, the match scoreassigned to two records is computed here as the number of identicallymatching DOB, Stad and SSN field values minus the number of mismatchedDOB, Stad and SSN fields, where if one record field value is blank, thenthe corresponding field is not taken into consideration for the matchscore, and where only match scores of at least one are considered.Alternately, a match score may be assigned according to any of thetechniques discussed in Section I above (e.g., in relation to Equations3-6) or in reference to FIGS. 5-7 of the First generation Patents AndApplications, or according to another technique. The match score may bea probability or a different measure of likelihood that the two recordsare related. Table VII.2 below is a non-limiting example of a matchtable (e.g., as produced at block 710) containing the match scoresassigned to each pair of records taken from Table VII.1. In the presentexemplary embodiment, only those record pairings with an assigned matchscore of at least one and where the left DID is greater than the rightDID may be further processed.

TABLE VII.2 Row Number Left DID Right DID Match Score Match Type 1 2 1 1Stad 2 3 1 1 Stad 3 4 2 2 Stad, DOB 4 5 1 2 Stad, SSN 5 5 3 2 Stad, DOB6 6 1 1 Stad 7 6 2 1 Stad 8 6 3 1 Stad 9 6 4 1 Stad 10 6 5 1 Stad

Row number 1 of Table VII.2 reflects that entries corresponding to DIDsof 1 and 2 in Table VII.1 share a common street address (Stad) only.Note that the entry corresponding to DID 1 in Table VII.1 has a blankDOB, while the entry corresponding to DID 2 in Table VII.1 has a blankSSN. Accordingly, for this embodiment, those fields do not count intothe match score assigned to the pair of records having DIDs of 1 and 2.Row number 3 in Table VII.2 reflects that records corresponding to DIDs4 and 2 in VII.1 share common street addresses (Stad) and DOBs.

At block 715, each record may be associated with a preferred record.Here, a preferred record associated with a given record is a record,which, when paired with the given record, has an assigned match scorethat is at least as great as any match score assigned to any record pairthat includes the given record. That is, an associated preferred recordof a given record is a record that, when paired with the given record,has a maximal assigned match score in comparison to a match scoreassigned to any other record pair comprising the given record. TableVII.3 below is a non-limiting example of a preferred record table (e.g.,as produced at block 720). That is, Table VII.3 contains preferredrecords associated with each record in Table VII.1, as determinedaccording to the matches of Table VII.2.

TABLE VII.3 Row No. DID RID DID Of Preferred Record 1 1 1 5 2 2 2 4 3 33 5 4 4 4 2 5 5 5 1, 3 6 6 6 1, 2, 3, 4, 5

By way of example, row number 1 of Table VII.3 reflects that the recordhaving RID 1 in Table VII.1 has as its associated preferred record therecord that appears in Table VII.1 with RID 5. This is because the matchscore assigned to any pair of records that includes the record with RID1 of Table VII.1 is no greater than 5. That is, 5 is the maximal matchscore assigned to any pair of records that includes the record with RID1 of Table VII.1. Note that it is possible for a record to have morethan one preferred record. Examples of such a situation appear in rows 5and 6 in Table VII.3. Row 5, for example, reflects that the match scoreassigned to the record pair consisting of the records with RIDs of 5 and1 is maximal, as is the match score assigned to the record pairconsisting of the records with RIDs of 5 and 3. Both match scores are 2,which is greater than any other match score assigned to a record pairthat includes the record with RID 5.

At least two relevant properties of records and their associatedpreferred records are apparent from an inspection of Table VII.3. First,as noted above, preferred records associated with a given record may notbe unique. This is the case for records with RIDs of 5 and 6.

Second, if A is a preferred record for record B, it is not necessarilythe case that B is a preferred record for record A. In mathematicalterms, the “preferred record” relation is not symmetric. For example, asseen above, the record with RID 6 has as one of its preferred recordsthe record with RID 2. However, the record with RID 2 does not have asits preferred record the record with RID 6. Thus, although the recordwith RID 6 has a preferred record with RID 2, the record with RID 2 doesnot have a preferred record with RID 6. In that sense, a “preferredrecord” may be an asymmetric, or one-way relationship.

At block 725, mutually preferred record pairs may be identified. Here, amutually preferred record pair is a pair of records, denoted A and B,such that A is a preferred record associated with B, and B is apreferred record associated with A. Note that, as discussed above, if Ais a preferred record associated with record B, then it is notnecessarily the case that B is a preferred record for A. However, themutually-preferred relationship is symmetric; that is, if A is amutually preferred record of B, then B is a mutually preferred record ofA. Table VII.4 below illustrates the mutually preferred record pairsderived from Table VII.3. Such a table may be generated by a method thatimplements block 720.

TABLE VII.4 Left DID Right DID 4 2 5 1 5 3

As seen in Table VII.4 and by way of example, records with DIDs of 4 and2 are mutually preferred records. This is because the record with DID 4has as its associated preferred record the record with DID 2, and therecord with DID 2 has as its associated preferred record the record withDID 4. Note further that the record with DID 6 does not appear in TableVII.4. This is because, although that record has several associatedpreferred records, no record has the record with DID 6 as its associatedpreferred record.

In some embodiments, when there are two mutually preferred pairs ofrecords, the pair with the highest match score may be retained forfurther processing. Note that the record pair with DIDs of 5 and 1 haveassociated with them a match score of two (2), as does the record pairwith DIDs of 5 and 3. In some embodiments, only mutually preferredrecord pairs with a left DID that is greater than the right DID andwhose right DID is the least possible may be considered. This techniqueserves to break such ties and avoid comparison between records that arealready linked. Table VII.5 illustrates this concept applied to TableVII.4.

TABLE VII.5 Left DID Right DID 4 2 5 1

Thus, Table VII.5 omits the DID pair 5, 3 because DID 5 is alreadypaired with DID 1, and 1 is less than 3.

At block 730, the mutually preferred record pairs may be linkedtogether. This may be done, by way of non-limiting example, byassociating the same DID with both mutually preferred records of thepair. Table VII.6 below illustrates how Table VII.1 may be altered toreflect the computations reflected in Tables 2-5.

TABLE VII.6 Row No. DID RID Fname Lname DOB Stad SSN 1 1 1 Mary James 7Main St. 123456789 2 2 2 Mary James 19970606 7 Main St. 3 3 3 Mary James19670923 7 Main St. 4 2 4 Mary James 19970606 7 Main St. 987654321 5 1 5Mary James 19670923 7 Main St. 123456789 6 6 6 Mary James 7 Main St.

Note that Table VII.6 reflects that the least DID may be inserted whenapplicable. For example, the record with RID 2 is linked to the recordwith RID 4. However, the least linking DID is 2; therefore, the DIDassociated with the record with RID 4 may be changed to the leastlinking DID, namely, 2. In alternate embodiments, the least DID may notbe used.

At block 735, the records undergo a propagation operation. Thisoperation may be essentially identical to the propagation operation thatmay occur between iterations of the techniques presented in Section II.That is, the records may be altered to include field values frommutually preferred records. Table VII.7 below illustrates suchalteration applied to Table VII.6. Note that the altered field valuesare italicized for illustrative purposes.

TABLE VII.7 Row No. DID RID Fname Lname DOB Stad SSN 1 1 1 Mary James19670923 7 Main St. 123456789 2 2 2 Mary James 19970606 7 Main St.987654321 3 3 3 Mary James 19670923 7 Main St. 4 2 4 Mary James 199706067 Main St. 987654321 5 1 5 Mary James 19670923 7 Main St. 123456789 6 66 Mary James 7 Main St.

As seen in Table VII.7, field values that were absent in records thatwere linked at block 730 are inserted. Table VII.7 illustrates theresults of a first iteration of an exemplary technique for identifyingand linking related records.

At block 740, blocks 705-735 may be iterated once more so as to furtheridentify and link related records. Iterations after the first iterationmay operate only on pairs of records that are not in the same entityrepresentation. Thus, block 705 may be repeated to assign match scoresto pairs of records that are not already in the same entityrepresentation. Table VII.8 illustrates a result of assigning matchscores to the records that appear in Table VII.7. The match scores areassigned as discussed above in reference to Table VII.1, and only thoserecord pairings with an assigned match score of at least one and wherethe left DID is greater than the right DID are illustrated.

TABLE VII.8 Row Number Left DID Right DID Match Score Match Type 1 3 1 2Stad, DOB 2 6 1 1 Stad 3 6 2 1 Stad 4 6 3 1 Stad

Next, in repeating block 715, preferred records for each entityrepresentation (blocks 705-730) may be calculated. The results aredepicted in Table VII.9 below.

TABLE VII.9 Row No. DID RID DID Of Preferred Record 1 1 1 3 2 2 2 6 3 33 1 4 6 6 1, 2, 3

Note that associated preferred records for records with RID of 4 and 5have previously been calculated when these records were linked toentities with DIDs 2 and 1, respectively.

Block 725 may be repeated to identify pairs of mutually preferredrecords. Table VII.10 below illustrates the application of this step tothe records reflected in Table VII.9.

TABLE VII.10 Left DID Right DID 3 1 6 2

Block 735 may then be repeated to alter the records by migrating datafield values between linked records. Note that the altered field valuesare italicized for illustrative purposes.

TABLE VII.11 Row No. DID RID Fname Lname DOB Stad SSN 1 1 1 Mary James19670923 7 Main St. 123456789 2 2 2 Mary James 19970606 7 Main St.987654321 3 1 3 Mary James 19670923 7 Main St. 123456789 4 2 4 MaryJames 19970606 7 Main St. 987654321 5 1 5 Mary James 19670923 7 Main St.123456789 6 2 6 Mary James 19970606 7 Main St. 987654321

At this point, the previously un-linked records depicted in Table VII.1are linked to two entity representations with respective DIDs of 1 and2. Iterating blocks 705-735 again will not result in further links. Thiscan be seen, for example, in attempting to repeat block 705 with therecords as they appear in Table VII.11. Upon such an attempt, it will beseen that no record pairs that are not already assigned to the same DIDhave non-zero match scores. That is, every record pair taken from TableVII.11 is either already linked or has an assigned match score thatindicates a low likelihood of being related. Accordingly, at this stage,the identification and linking process of FIG. 7 is complete.

According to an exemplary embodiment of a technique of this section, inan electronic database comprising a first plurality of records, each ofthe first plurality of records comprising a plurality of data fields,each of the data fields capable of containing a field value, a method ofidentifying and linking related records is presented. The embodimentincludes assigning to each pair of records from the first plurality ofrecords a match score, the match score reflecting a probability that thepair of records is related. The embodiment includes determining, foreach record from a second plurality of records, at least one associatedpreferred record, where the first plurality of records includes thesecond plurality of records, where a match score assigned to a givenrecord together with its associated preferred record is at least asgreat as a match score assigned to the record together with any otherrecord in the first plurality of records. The embodiment includesidentifying mutually preferred pairs of records from the secondplurality of records, each mutually preferred pair of records consistingof a first record and a second record, the first record consisting of apreferred record associated with the second record and the second recordconsisting of a preferred record associated with the first record. Theembodiment includes, for at least one mutually preferred pair of recordsconsisting of a third record and a fourth record, linking the thirdrecord to the fourth record. The embodiment includes allowing a user toretrieve information from at least one of the third record and thefourth record.

Various optional features of the above exemplary embodiment include thefollowing. The embodiment may include that each preferred recordassociated with a given record includes a record that, when paired withthe given record, has a maximal assigned match score in comparison tomatch scores assigned to other record pairs comprising the given record.The embodiment may include, for at least one mutually preferred pair ofrecords consisting of a fifth record and a sixth record, altering atleast one field value from the fifth record based on at least one fieldvalue from the sixth record. The embodiment may include that the matchscore reflects a number of data field entries common to the pair ofrecords.

Another optional feature of the above exemplary embodiment includes thefollowing. The embodiment may include, prior to the step of linking,assigning to each pair of records from a third plurality of records amatch score, the match score reflecting a probability that the pair ofrecords is related, where the second plurality of records includes thethird plurality of records, determining, for each record from a fourthplurality of records, at least one associated preferred record, wherethe third plurality of records includes the fourth plurality of records,where a match score assigned to a given record together with itsassociated preferred record is at least as great as a match scoreassigned to the record together with any other record in the thirdplurality of records, and identifying mutually preferred pairs ofrecords from the fourth plurality of records, each mutually preferredpair of records consisting of a fifth record and a sixth record, thefifth record consisting of a preferred record associated with the sixthrecord and the sixth record consisting of a preferred record associatedwith the fifth record.

Various optional features of the above exemplary embodiment include thefollowing. The embodiment may assign a match score assigned to a pair ofrecords as determined by comparing data field entries of the pair ofrecords. The embodiment may include comparing data field entriesincludes comparing only a portion of data fields common to the pair ofrecords. The embodiment may assign a match score assigned to a pair ofrecords as calculated based at least on entries in at least one datafield common to each record of the pair. The embodiment may include thatthe database includes a fifth record and a sixth record, where the fifthrecord is an associated preferred record of the sixth record and wherethe sixth record is not an associated preferred record of the fifthrecord.

The above described embodiments are exemplary only and are not intendedto limit the scope of the inventions disclosed herein.

VIII. Automated Selection of Generic Blocking Criteria

Embodiments of this technique may be implemented in their own iterativeprocess, incorporated into a non-iterative process, or incorporated intoan iterative process such as described above in Section II. Moreparticularly, embodiments of the technique of this section may be usedto generate blocking criteria for use in a linking process betweeniterations.

In certain embodiments of the technique of this section, fieldprobabilities associated with the fields (which may be calculatedaccording to an embodiment shown above) may be used to create one ormore blocking criteria. A blocking criteria may use a first algorithm oralgorithms on a universe of records to create a potential candidate listfor linking records. The potential candidate list created by the firstalgorithm or algorithms may be a subset of the universe of records. Asecond algorithm or algorithms may then be used on the recordsassociated with the potential candidate list to search for links betweentwo or more records. The first algorithm or algorithms may becomputationally faster and/or relatively more inaccurate than the secondalgorithm or algorithms, so that a faster, yet relatively less accurate,algorithm may be used to create a subset of the universe of records, andthen a slower and more accurate algorithm may be used to link associatedrecords from the subset. An example of one type of blocking may be foundin U.S. Pat. No. 7,152,060 to Borthwick, et al. (“Borthwick '060”).

One possible input into the embodiment may be a maximum number ofrecords to be returned in a blocking operation, or a maximum block size.The maximum block size may be an absolute number, for example a maximumblock size may be 100 records, or the maximum block size may be arelative number, for example ten percent of the records in a databasetable.

One possible output from the embodiment may be one or more blockingcriteria. Each blocking criteria may be a list of fields that should beequal among two or more records so that, for example, a search of theuniverse of records specifying that the blocking criteria fields areequal returns a subset of the universe of records of approximately themaximum block size. In an embodiment, the blocking criteria fields maybe equal and non-null (e.g., the field must contain data). The subset ofthe universe of records may be smaller than the specified maximum blocksize. If the embodiment cannot create a blocking criteria that wouldreturn a subset of the universe of records equal to or smaller than thespecified maximum block size, the embodiment may return an error messagefor processing, or may return a blocking criteria yielding a block sizeas close to the maximum block size as possible. The blocking criteriamay be generated using an exhaustive search of all of the fields andfield probabilities that may yield a number of records at or below themaximum block size, or by using a faster but non-exhaustive search ofthe various combinations of fields and associated field probabilities.

As discussed above, fields in the database may have field probabilitiesp_(f) associated with them, so that, for example, more significance maybe shown to records that match on a field with a small or lowprobability (e.g., the probability that two records denoting the sameperson may have identical social security numbers may be high, and theprobability that two records denoting different persons, where bothrecords have the same social security number, may be low) than may beshown to records that match on a field with a large or high probability(e.g., there may be a higher probability that several people all havethe identical first name, and the fact that two records may not haveidentical first name fields may not be a definitive indicator that therecords are not associated with the same person). Discussed above, onaverage, p_(f) times the number of records in the database provides theaverage cohort size for field values in field f. That is, p_(f) timesthe number of records in the database (respectively, the number ofentity representations in the database) gives the average number ofrecords (respectively, the average number of entity representationscontaining records) containing the same field value in field f. In otherwords, p_(f) times the number of records under consideration providesthe average size of each field value cohort in field f Moreover, p_(f)may be independent of field value. A p_(f) value produced by anyiteration of a Section II embodiment may be used, and for the purposesof this embodiment, any field probability by any iteration of a SectionII embodiment may be referred to as p_(f).

Note that certain embodiments of the technique of this section mayinclude multiplication of a number of probabilities (e.g., multiplevalues of p_(f)). As discussed in Section I, for computationalconvenience, the probabilities may be converted to weights and operatedupon in the log domain. More particularly, as discussed in Section I,products of probabilities may be essentially isomorphic to sums oflogarithms of such probabilities. Accordingly, although this section isdiscussed in terms of multiplication within the probability domain, thecalculations may be implemented in the log domain with addition in placeof multiplication where appropriate. Conversion between logs andprobabilities and back may occur at various points in certainembodiments of the present technique.

The algorithm used to find the blocking criteria may use as an input thefield probabilities associated with one or more of the fields in thedatabase. The algorithm may also use other inputs to determine blockingcriteria. In one embodiment, the knapsack algorithm may be used todetermine blocking criteria, the operation and implementation of whichis known to ordinary skill in the art. The knapsack algorithm may beused to determine one or more possible blocking criteria, given thefield probabilities associated with one or more of the fields in thedatabase. A “greedy” algorithm may be used to choose fields with thehighest w_(f)(accordingly, the lowest p_(f)) values first. Otheralgorithms may be used to create one or more blocking criteria, forexample an exhaustive search of the fields and associated fieldprobabilities may be conducted to determine blocking criteria.

In one embodiment, the identification algorithm used to find theblocking criteria may include creating a list of the fields and thefield probabilities associated with each field, and sorting the fieldslowest to highest according to the field probabilities. Theidentification algorithm may then choose the first field in orderedprobability list, and may add that field to the blocking criteria. Theidentification algorithm may then move down the list of fields, addingsubsequent fields to the blocking criteria until the number of matchingrecords (respectively, entity representations) associated with theblocking criteria is equal to or less than the maximum block size input.The number of associated records (respectively, entity representations)may be computed as a product of the relevant p_(f) values and the totalnumber of records (respectively, entity representations). Theidentification algorithm may then re-create the list of the fields andthe field probabilities, and choose another starting point on the listto create another blocking criterion.

Consider a database having 100 records with p_(f) values for the fieldsreflected in the table below.

Field Name p_(f) First Name 0.4 Last Name 0.3 Address 0.2 City 0.5 State0.6 Zip Code 0.4 SSN 0.1

Also consider that the embodiment is given a maximum block size of six.From the above table, none of the individual field probabilities may beapplied to the database to return only six records that may beassociated with one record in the database. However, insisting that twoor more fields match generally reduces the size of the block. Thus, twoor more fields may be used to narrow the total number of recordsreturned in batch operation on each record, so that six or fewer recordsare identified for each of the records in the database. For example, the{SSN, Address} combination of fields may be used, so that given a recordin the database, the blocking operation of the embodiment may return anumber of records equal to or less than the maximum block size that haveidentical or similar fields.

Turning to FIG. 8A, in block 810, the embodiment may create or use anexisting list of fields and associated field probabilities. In block815, the embodiment may sort the table of fields and p_(f) values sothat fields with low p_(f) values are listed first. The embodiment mayalso calculate a combined probability total up the list, that is, apotential remaining probability (“PRP”). For example:

Field Name p_(f) PRP SSN 0.1 0.00288 Address 0.2 0.0144 Last Name 0.30.048 First Name 0.4 0.12 Zip Code 0.4 0.3 City 0.5 0.6 State 0.6

The PRP field for the City row may be the p_(f) value of the Statefield, the PRP field for the Zip Code row may be the PRP value of theCity row multiplied by the p_(f) value of the City field, the PRP fieldfor the First Name row may be the PRP value of the Zip Code rowmultiplied by the p_(f) value of the Zip Code field. This process mayiterate until the PRP values have been calculated for the orderedprobability list.

The identification algorithm of the embodiment may order the fields sothat the fields with the lowest probabilities are listed first, and maystart with the SSN field, since that field has the lowest p_(f) value.Shown in block 820, the identification algorithm may then construct asearch tree according to the data provided in the present example, whichis partially shown in FIG. 8 b.

Turning again to FIG. 8A, in block 825, the search tree may be used todetermine if any of the search tree “branches” may be disregardedbecause, for example, they cannot yield a maximum block size at or belowthe specified maximum block size. In some embodiments, alpha-betapruning on the search tree may be performed to remove one or more“branches” that are not able to meet the specified maximum block size.The PRP values in the table, shown above, may be the product of p_(f)values of all of the fields that are lower on the ordered probabilitylist, and may be used to calculate an estimated overall probability. Forexample, if the embodiment attempted to calculate the estimatedprobability of records in a {SSN, Last Name, First Name, Zip Code, City,State} combination, the embodiment may find the p_(f) value of the SSNfield and multiply that probability by the PRP value for the Addressfield, yielding an estimated probability of (0.1*0.0144=0.00144), withina maximum block size value of 6 records in 100 overall records, or 0.06.The embodiment may also attempt to calculate the estimated probabilityof records in a {Zip Code, City, State} combination. In this example,the embodiment may take the p_(f) value of the Zip Code field andmultiply that probability by the PRP value for the Zip Code field,yielding an estimated probability of (0.4*0.3=0.12), outside of amaximum block size value of 0.06. Since the estimated probability isgreater than the maximum block size value, all of the sub-combinationsof those fields also cannot be less than the maximum block size value.The embodiment may then “prune” all of the combinations of {Zip Code,City}, {Zip Code, State}, and {Zip Code, City, State} from the searchtree.

In one example, the embodiment may begin with the SSN field (p_(f)=0.1).Since the SSN field may not, by itself, yield a block size equal to orless than the six block maximum block size, the embodiment may choosethe field with the next highest probability value (Address, p_(f)=0.2),which when combined with the State value may yield a percentage ofrecords of 0.02, which may yield a block size less than the specifiedmaximum block size of six in a 100 record database table.

In block 830, the embodiment may add a set of fields to the list ofblocking criteria. In the present example, the embodiment may add thecombination of {SSN, Address} to the list of blocking criteria. In block835, the embodiment may then remove the last field added to thecombination (in the present example, the Address field), and may thenattempt to add the next field in the ordered probability list (LastName, p_(f)=0.3). The union of the SSN and Last Name fields may yield apercentage of records of 0.03, which would yield a block size less thanthe specified maximum block size of six. The embodiment may add thecombination of {SSN, Last Name} to the list of blocking criteria, andmay then remove the last field added to the combination (in this case,the Last Name field), and may then attempt to add the next field in theordered probability list (First Name, p_(f)=0.4). The union of the SSNand First Name fields may yield a percentage of records of 0.04, whichwould yield a block size less than the specified maximum block size ofsix. The embodiment may add the combination of {SSN, First Name} to theblocking criteria list, and may then remove the last field added to thecombination (in this case, the First Name field).

The embodiment may then attempt to iterate through the remaining fieldsin a similar way, and if the union of the remaining fields with the SSNfields combination yields a percentage of records equal to or less than0.06 (6 records in 100 total records), the embodiment may add the newcombination to the blocking criteria list. When the remaining fieldshave been checked, the embodiment may remove the SSN field from thecombination, and may add the next field in the ordered probability list(in this case, the Address field).

The embodiment may choose to add the Last Name field to the {Address}combination. The union of {Address, Last Name} may yield a percentage ofrecords of 0.06, equal to the maximum block size. The embodiment may addthe {Address, Last Name} combination to the blocking criteria list, andmay disregard further combinations with the {Address, Last Name}combination, since the minimum block size is met with the {Address, LastName} combination. The embodiment may remove the Last Name field fromthe combination, and may add the next field in the ordered probabilitylist (First Name, p_(f)=0.4).

The combination of the Address and First Name fields may not combine toyield a block size equal to or less than the six block maximum blocksize. The embodiment may choose the field with the next highestprobability value (Zip Code, p_(f)=0.4). If two or more embodimentsshare the same p_(f) value, the embodiment may choose one of the fieldsbased on other criteria. The combination of the Address, First Name, andZip Code fields may have a percentage of records of 0.03, which mayyield a block size less than the specified maximum block size of six ina 100 record database table. The embodiment may then add the {Address,First Name, Zip Code} combination to the blocking criteria list, and mayremove the last field added (in this case, the Zip Code field).

Note that is some embodiments, the actions of block 825 may occurconcurrently with the actions of blocks 830 and 835. That is, in suchembodiments, the search tree may be traversed by either pruning (block825) or adding to the blocking criteria (blocks 830 and 835), dependingon the individual status of a node in the tree.

In block 840, the embodiment may iterate through the list of fields,attempting to create combinations of fields that yield a percentage ofrecords equal to or less than the maximum block size. The embodiment mayalso attempt to create combinations of fields that yield a percentage ofrecords that are approximately equal to the maximum block size. Forexample, the embodiment may create combinations of fields that are notmore than one percent more than the desired block size, or not more thanfive percent more than the desired block size.

The data within the fields specified above may be equal in the targetrecord and each of the records returned in the block, and the number ofrecords in the block may be equal to or less than the specified maximumblock size.

The creation of blocking criteria may be applicable to the entiredatabase, so that the blocking criteria may be used to create blocks ofrecords for any query in the database. In other words, the blockingcriteria generated may be a “generic” blocking criteria. The genericblocking criteria may be applicable to create blocks of records that maybe associated with each of the records in the database. The embodimentmay be used to create blocking criteria for a batch comparison of all orsubstantially all of the records in the database. The blocking criteriamay thus be used to associate the records into one or more blocks, sothat a query on any target record in the database may begin with theblocking criteria to narrow the universe of records in the database to amore manageable block of records that may be more carefully scrutinizedfor potential links to the target record. Assuming a universe of onehundred billion records, a batch comparison operation attempting tomatch all records to all other records may be prohibitive. If theembodiment can use a blocking criteria to narrow the universe of recordsto one hundred or one thousand records that may be potentiallyassociated with each record in the database, the time and computationrequired for a batch comparison operation may be reduced.

The embodiment of the invention may create a generic blocking criteriawhich may be used to associate each of the records in the database toother records in the database. The generic blocking criteria may be usedto complete a batch comparison or a batch linking operation withinrecords in the database. The embodiment may be used to generate ageneric blocking criteria, applicable to all of the records of thedatabase regardless as to specific field values. Blocks of recordshaving the same field value(s) in the field(s) of the blocking criteriawill generally be the appropriate size irrespective of the specificfield value(s). Thus, the blocking criteria may be used to generateblocks of suitable size without regard to any particular field value(s),and so the embodiment may produce an optimal or near-optimal set ofcriteria to compare all records within the database to all other recordsthat may be matched. This feature, for example, distinguishes theembodiment from Borthwick '060. Borthwick '060, in contrast, generatesblocking criteria that are specific and suited only for individual,user-defined queries against a database. Many other features distinguishthis embodiment from Borthwick '060. For example, and withoutlimitation, Borthwick '060 creates blocking criteria only in response toa user-defined query against a database. Some of the present embodimentsmay create generic blocking criteria that may be used to narrow therecords that may be associated with each of the fields in the database.Other features of the embodiment also distinguish the embodiment fromBorthwick '060.

According to an exemplary embodiment of a technique of this section, amethod of creating blocking criteria based on a maximum block size ispresented. The embodiment includes calculating one or more fieldprobabilities for the one or more fields in the database. The embodimentincludes determining one or more fields which must be equal to create ablock size equal to or less than the maximum block size. The embodimentincludes grouping records in the database into one or more blocks byapplying the blocking criteria.

Various optional features of the above embodiment include the following.The embodiment may include that the blocking criteria generated eachcreate a partition for each of the records in the database. Each suchpartition may be different. The embodiment may include the additionalstep of applying the blocking criteria to each record in the database.

The above described embodiments are exemplary only and are not intendedto limit the scope of the inventions disclosed herein.

IX. Automated Calibration of Negative Field Weighting Without the Needfor Human Interaction

A technique for calculating and calibrating negative field weights ispresented here. This technique may be implemented in an iterationaccording to an embodiment of the technique of Section II. Further, thistechnique may be utilized in matching formulas such as those discussedin the context of Equations 3-6 in Section I.

An exemplary, non-limiting embodiment of the present technique may beimplemented as follows. In particular, the exemplary embodiment isdiscussed in the context of the matching formulas discussed in Section Iand the exemplary iteration discussed in Section II. In general, theexemplary embodiment proceeds according to an embodiment of Sections Iand II, but with certain terms given particular negative values. Thus,the following discussion should be viewed as a discussion of how tomodify the techniques presented in Sections I and II. Except asindicated otherwise, either explicitly or implicitly, the presentembodiment may proceed as an embodiment of the techniques of Sections Iand II.

FIG. 9 is a flowchart depicting an exemplary embodiment according tothis section. The embodiment may begin in a preliminary linkingoperation that occurs after the first iteration. As discussed there, thepreliminary linking operation may utilize a matching formula, such asthose discussed in Section I in reference to Equations 3-6. Theembodiment modifies Equations 3-5 by allowing certain terms in suchmatching formulas to be negative in the event of a non-match in one ormore given fields. The technique is discussed in the context of Equation3, however, it may be applied in the context of Equations 4 or 5. Forconvenience, Equation 3 is reproduced below:

${S\left( {r_{1},r_{2}} \right)} = {\sum\limits_{f}\; {p_{f}{w_{f}.}}}$

At block 905, a pair of records r₁, r₂ are considered. At block 910, itis determined whether there is a match between the records in field f.If the field values match between records r₁ and r₂ in field f, then theterm p_(f)w_(f) in Equation 3 may be determined as discussed above inSections I and II at block 915. For non-matches, according to theexemplary embodiment under discussion, the term p_(f)w_(f) may behandled differently at block 920. Specifically, for a non-match in fieldf of records r₁ and r₂ as calculated during the preliminary linkingoperation after the first iteration, the term p_(f)w_(f) may be set tothe negative of w_(f)(i.e., −w_(f)). The sum may otherwise be computedas discussed in Section I, and the records r₁ and r₂ may be linked basedon whether S(r₁, r₂) exceeds a threshold, which may be determinedaccording to Equation 6. At block 925, the iteration may proceed byutilizing p_(f)w_(f).

For iterations after the first iteration, the exemplary embodiment mayproceed as follows. Specifically, the embodiment may continue bymodifying the linking operation that occur after the second andsubsequent operations. Note that after the first iteration, the databasegenerally contains entity representations instead of solely unlinkedrecords. As with the first iteration, the present embodiment affects theterms in the matching formula used to determine links between records.Specifically, for two records r₁ and r₂ provided at block 905 and for agiven field f, if the field values in the given records match asdetermined at block 910, then the term p_(f)w_(f) may be calculated asdiscussed above in Sections I and II at block 915. If the field valuesdo not match, the term p_(f)w_(f) may be modified as follows at block920. First, a count may be made of entity representations that have theproperty that they include two linked records that have different fieldvalues in field f. That count may be divided by the total number ofentity representations. The resulting ratio may be subtracted from one(1), negated, and multiplied by w_(f). The resulting term may be used inplace of p_(f)w_(f). These operations may be conducted relative to theentity representations that exist during the relevant linking operation(after the relevant iteration). A calculation of one or more of thecount, the ratio, the ratio subtracted from one, and the ratiosubtracted from one and negated may be made at the time of the firstiteration (e.g., before, during or after calculating the weights w_(f))and, for later use, stored in, for example, a lookup table or in anextra field added to one or more records. Thus, for two given records r₁and r₂ and for a given field f, if the field values in the given recordsdo not match, then the term p_(f)w_(f) may be calculated as, by way ofnon-limiting example:

$\begin{matrix}{{p_{f}w_{f}} = {- {{w_{f}\left( {1 - \frac{K_{f}}{K}} \right)}.}}} & {{Equation}\mspace{14mu} 29}\end{matrix}$

In Equation 29, K_(f) represents the number of entity representationsthat include two records with different field values in field f, and Krepresents the total number of entity representations. In someembodiments, the term K is determined as the number of entityrepresentations that include at least two different records. Note that Kand K_(f) may be computed during an iteration according to thetechniques set forth in, for example, Section II. More particularly,these terms may be computed as part of such an iteration and stored foruse according to a technique of this section. Equation 29 provides aformula for terms in the event of a non-match in field f. At block 925,the terms provided by Equation 29 may be included in any of Equations3-5 in order to determine, in conjunction with Equation 6, whetherrecords should be linked.

An alternate formula to that provided by Equation 29 is discussedpresently. Equation 30, presented below, may be used in exactly the samecircumstances as those discussed above in relation to Equation 29. Inthe case of a non-match in field f, the term p_(f)w_(f) may becalculated as, by way of non-limiting example:

$\begin{matrix}{{p_{f}w_{f}} = {{- \log}{\frac{K_{f}}{K}.}}} & {{Equation}\mspace{14mu} 30}\end{matrix}$

In Equation 30, the terms K_(f) and K represent the quantities discussedabove in reference to Equation 29. Thus, in the case of a non-matchbetween records in field f, the term p_(f)w_(f) in any of Equations 3-5may be set as provided by Equation 30.

The above described embodiments are exemplary only and are not intendedto limit the scope of the inventions disclosed herein.

X. Statistical Record Linkage Calibration for Multi Token Fields Withoutthe Need for Human Interaction

Embodiments of this technique may be implemented in their own iterativeprocess or incorporated into an iterative process as described above inSection II.

In some embodiments, the techniques of this section provide one or moreweights (“blended weights”), which may be used in a record matchingformula (e.g., Equations 3-5) to scale probabilities (e.g., p_(f) orp_(f)) that two records contain nearly matching field values in a fieldthat typically contains multiple tokens. Examples of such fields,referred to as “multi token fields,” include business name fields,street name fields, free text fields, etc. (The term “token” encompassesany part of a field value.) A near match of field values containingmultiple tokens may be indicated by exact matches between some or alltokens, near matches between some or all tokens, or a combination ofboth. A near match between a pair of individual tokens may be determinedaccording to any of the various near match metrics disclosed herein orotherwise, including SOUNDEX, edit distance, etc. Some embodimentsdetermine a probability (referred to as a “token probability”)associated with each separate token that makes up a field value, andconvert such probabilities to weights (referred to as “token weights”).Further, the entirety of each multi token field value in each multitoken field may have an associated to it a field value weight accordingto the techniques discussed in Section II.

Accordingly, an entire field value may have an associated field valueweight, and the tokens that make up the entire field value may each haveassociated token weights. These and other weights may be mathematicallyblended to arrive at a “blended field value weight.” Thus, in someembodiments, a weight associated with a multi token field value for usein a matching formula (e.g., Equations 3-5) may be determined bymathematically blending the weight associated with the entire fieldvalue and the weights associated with each constituent token. Moregenerally, a “blended weight” used in a matching formula to determinewhether to link two records, where the blended weight is associated witha multi token field, may determined by mathematically blending two ormore of the following: any of the weights associated with the entirefield values in the multi token fields in each of the records undercomparison, and any of the weights associated with each token thatappears in the field values in the multi token fields in each of therecords under comparison. Thus, each multi token field value may haveassociated to it a blended weight, each pair of multi token field valuesmay have an associated blended weight, and each of these blended weightsmay be used in making linking decisions as discussed above in Section I.

Certain embodiments associate a probability to each multi token field,independent of any particular field value. For a given multi tokenfield, the associated probability may be computed as a weighted averageof the probabilities associated with each individual token that mayoccur in the multi token field. These field probabilities calculated bycertain embodiments may be converted to field weights and used in makingrecord linking decisions. Such decisions may take into account some orall of the fields common to the records. In this technique, knowledge ofthe common field values may be not required. Further, this techniqueproduces accurate results for any two records, regardless as to thecontents of their fields.

In some embodiments, the field probabilities may be used for qualityassurance purposes. For example, the field probabilities may be used toquantitatively monitor the diversity of tokens that appear in aparticular multi token field. A relatively large field value associatedwith a multi token field can indicate that the multi token fieldcontains a large number tokens with relatively large associated fieldvalue weights. (A relatively large field value weight may indicate thatthe associated field value is relatively rare.) Such a relatively largefield value weight may indicate that a number of records with junkentries in the multi token field have entered the database.

FIG. 10 is a flowchart depicting an exemplary embodiment according tothis section. The present embodiment may be implemented in conjunctionwith an embodiment of the techniques of Section II. For purposes ofillustration rather than limitation, the present embodiment will bediscussed in reference to exemplary records r₁, r₂ and r₃ reflected inthe table below. Thus, Table X.1 below reflects a portion of a database.

TABLE X.1 Record DID Business Name Zip Code Phone Number r₁ 1 Joe's LawnFurniture 22222 (703) 555-1000 Corporation r₂ 2 Abe's Lawn Furniture33487 (561) 555-1234 Corporation r₃ 3 Joe's Furniture Corporation 22222

A visual inspection of these three records reveals that records r₁ andr₃ are likely for the same individual, in this case, a company thatapparently deals in lawn furniture. However, naïvely using an editdistance metric, for example, to compare the Business Name field valueswould indicate a closer match between r₁ and r₂ than between r₁ and r₃.That is, the field value “Joe's Lawn Furniture Corporation” is closer to“Abe's Lawn Furniture Corporation” than it is to “Joe's FurnitureCorporation” when using, for example, an edit distance metric to gaugecloseness. However, the field values “Joe's Lawn Furniture Corporation”and “Joe's Furniture Corporation” are more likely to correspond to thesame individual than the field values “Joe's Lawn Furniture Corporation”and “Abe's Lawn Furniture Corporation”. Certain embodiments of thetechniques discussed in this section provide a way to compare multitoken field values in a way that provides better field value weights forfield values in multi token fields than those that might be availablefrom naïvely using edit distance. Note, however, that certainembodiments of the technique of this section may utilize edit distancemetrics in a way that improves upon the prior art.

At block 1000, the exemplary embodiment begins a first iteration byselecting one or more multi token fields and then, for each such field,constructing a table at block 1005 that contains a record for each tokenthat appears in any record in the selected field. Such a table will bereferred to as a “token table.” Token tables may include duplicates, forexample, if the same token appears more than once in the same record orin different records. The token table may further contain definitiveidentifiers associated with each token. That is, the token table mayassociate to each token the DID of the original record in which thetoken appeared. A first iteration token table corresponding to BusinessName field of the records in Table X.1 appears below; however, thepresent technique may be applied to any number of multi token fields.

TABLE X.2 DID Token 1 Joe's 1 Lawn 1 Furniture 1 Corporation 2 Abe's 2Lawn 2 Furniture 2 Corporation 3 Joe's 3 Furniture 3 Corporation

Note that, as Table X.1 depicts a portion of a database, so too doesTable X.2.

At block 1010, the first iteration may proceed to compute token fieldvalue probabilities and token field value weights for the records in thetoken table (Table X.2). That is, the first iteration may proceed bycalculating token probabilities and token weights. These may be computedusing any of the techniques disclosed herein. For purposes ofillustration, the techniques of Section II may be applied to a tokentable, such as Table X.2.

Thus, for each token, the first iteration may proceed by determining thenumber of tokens with unique DIDs that are present in the token table.That is, the first iteration counts the number of records in the tokentable that include a particular token, counting multiple tokens thatappear in the same original field as one. At this point, every token hasan associated count. These counts may be then divided by the totalnumber of different DIDs in the token table, yielding tokenprobabilities. (In the first iteration, the number of different DIDs maybe the number of records in the original database, a portion of which isreflected in Table X.1.) Thus, at the end of the first iteration, eachtoken together with the field in which it originally appeared may beassociated with a token probability, which may be calculated as thenumber of records with unique DIDs in the token table that include thetoken, divided by the total number of unique DIDs in the token table. Informal terms, the token probability may be calculated as, by way ofnon-limiting example:

$\begin{matrix}{{p_{f,t}(1)} = {\frac{c_{f,t}}{c}.}} & {{Equation}\mspace{14mu} 31}\end{matrix}$

In Equation 31, the term p_(f,t)(1) represents the first iteration tokenprobability associated with original field f and token t. The termc_(f,t) represents the number of records with unique DIDs that appear inthe token table that include token t, and the term c represents thetotal number of different DIDs that appear in the token table.Accordingly, a given token probability produced by the first iterationmay be a probability that a record randomly chosen from the (original)database contains the given token in the associated field. The fieldvalue probabilities may be converted to field value weights accordingto, by way of non-limiting example:

w _(f,t)(1)=−log p _(f,t)(1).   Equation 32

Thus, at the end of the first iteration, each token and the originalfield in which it appears may be associated with a token weight, each ofwhich may be calculated from a corresponding token probability.

Once the first-iteration token probabilities and token weights arecalculated, the token table may be modified by adding columns for one orboth of field value probabilities and field value weights. An exemplarysuch token table, based on Table X.2, appears below.

TABLE X.3 DID Token Token Weight 1 Joe's 15 1 Lawn 10 1 Furniture 12 1Corporation 4 2 Abe's 20 2 Lawn 10 2 Furniture 12 2 Corporation 4 3Joe's 15 3 Furniture 12 3 Corporation 4

Various techniques may be used to store the field value probabilitiesand field value weights for later use. Such parameters may be stored ina table, such as one as represented by Table X.3. By way of non-limitingexample, field value weights may be stored in fields added to records inwhich the associated field values appear. In some embodiments, one orboth of token probabilities and token weights may be stored in fieldsappended to records, while one or both of associated field probabilitiesand field weights may be stored in one or more lookup tables. As anothernon-limiting example, the token weights may be stored by modifying theoriginal field values in the original records in which the tokensappear. More particularly, the token weights may be inserted before orafter their associated token in the original field in which the tokenappears. The table below illustrates an application of this technique tothe original records r₁, r₂ and r₃, appending the token weights to theassociated token.

TABLE X.4 Zip Record DID Business Name Code Phone Number r₁ 1 Joe's 15Lawn 10 Furniture 22222 (703) 555-1000 12 Corporation 4 r₂ 2 Abe's 20Lawn 10 Furniture 33487 (561) 555-1234 12 Corporation 4 r₃ 3 Joe's 15Furniture 22222 12 Corporation 4

At block 1015, also during the first iteration, the records of theentire database (or portion thereof) may undergo a separate firstiteration to generate match weights and probabilities. That is, thedatabase at large (or portion thereof) may undergo a first iterationaccording to, for example, the techniques of Section II (e.g., usingEquations 7 and 8). Such an iteration may generate field valueprobabilities and field value weights for the entirety of each multitoken field value. Thus, subjecting the database to a first iterationmay generate field value weights for each of the entire multi tokenfield values.

The first iteration may proceed to generate field probabilities andfield weights associated with each multi token field. Such parametersmay be generated using the techniques described in Section II (e.g.,Equations 9 and 10) and stored for later use according to any of thestorage techniques disclosed herein.

Thus, the first iteration applied to the database may generate fieldvalue weights for each field value that appears in the Business Namefield, as well as a field weight for that field (e.g., using Equations7-10). These weights may be stored as discussed in Section II.Alternately, or in additional, the weights may be stored in the originalrecords as part of the field value itself. By way of non-limitingexample, suppose the weight given to the field value “Joe's LawnFurniture Corporation” is 35, the weight for field value “Abe's LawnFurniture Corporation” is 38, and the weight given to field value “Joe'sFurniture Corporation” is 27. These weights may be stored, by way ofnon-limiting example, as prefixes to the field values that appear in theBusiness Name field. A table illustrating this technique appears below.

TABLE X.5 Record DID Business Name Zip Code Phone Number r₁ 1 35 Joe's15 Lawn 10 22222 (703) 555-1000 Furniture 12 Corporation 4 r₂ 2 38 Abe's20 Lawn 10 33487 (561) 555-1234 Furniture 12 Corporation 4 r₃ 3 27 Joe's15 Furniture 22222 12 Corporation 4

At block 1020, the exemplary embodiment under discussion may proceed togenerate blended field value weights from the token weights and thefield value weights for the entire field values that appear in the multitoken field. These blended field value weights may be calculated withrespect to pairs of records. That is, given a pair of records, a blendedfield value weight may be generated based on the token weights, thefield value weights for the entire multi token field, the number ofmatching tokens between the pair of records in the given multi tokenfield, and the number of non-matching tokens between the pair of recordsin the given multi token field. Thus, a blended field value weight maybe associated with a pair of field values that appear in a multi tokenfield in two records. The blended field value weights may be used tomake linking decisions between pairs of records according to, forexample, Equations 3-6. The blended field weights may be generated andstored, or generated on the fly as part of the linking decision process.

Blended field value weights may be computed according to a variety oftechniques. The following provides exemplary techniques for blending thetoken weights and field value weights. For a pair of records r_(i) andr_(j) that share a multi token field, where i and j represent indexesfor any two records, any of the following equations may be used tocalculate a blended weight associated with the multi token field valuesin the two records in the shared multi token field.

$\begin{matrix}{w_{i,j} = {{\max \left( {w_{i},w_{j}} \right)} \times \frac{2\; M}{{2\; M} + N}}} & {{Equation}\mspace{14mu} 33} \\{w_{i,j} = {{\min \left( {w_{i},w_{j}} \right)} \times \frac{{2\; M} - N}{{2\; M} + N}}} & {{Equation}\mspace{14mu} 34} \\{w_{i,j} = {{\max \left( {w_{i},w_{j}} \right)} \times \frac{{2\; M} - N}{{2M} + N}}} & {{Equation}\mspace{14mu} 35} \\{w_{i,j} = {\min \left( {w_{i},w_{j},M} \right)}} & {{Equation}\mspace{14mu} 36}\end{matrix}$

In the Equations 33-36, the term w_(i,j) denotes the blended field valueweight associated with records r_(i) and r_(j) and a selected sharedmulti token field (and, of course, the multi token field values thatappear in such field in the two records), w_(i) denotes the field valueweight associated with the entire multi token field value in theselected multi token field of r_(i), w_(j) denotes the field valueweight associated with the entire multi token field value in theselected multi token field of r_(j), the terms min( ) and max( ) denotethe minimum and maximum operators, respectively, the term M denotes thesum of the token weights for tokens that match between records r_(i) andr_(j), and the term N denotes the sum of the token weights for tokensthat do not match between records r_(i) and r_(j).

Equations 33-36 may be suitable for generating blended weights with avariety of different properties. Equation 33 may be suitable for generalmulti token fields, such as business names. Equation 34 may be suitablefor multi word descriptions of complex entities, such as departments inbusinesses or hospitals. Equation 35 may be viewed as a scaled andbiased version of Equation 33 that may be used to handle multi tokenfields in which it is unlikely that two or more field values may be usedto describe the same thing. Equation 36 may be used for multi tokenfields that contain lists, such as names of co-inventors on patents.Although certain properties and suitabilities are discussed in thisparagraph, any of Equations 33-36 ay be used to generate blended fieldvalue weights for any multi token fields, not limited to those discussedin this paragraph.

The following provides calculations according to Equations 33-36 withrespect to the pair of records r₁ and r₂ taken from Table X.1 and theshared multi token Business Name field. Note that, in this example forrecords r₁ and r₂, the terms w₁ and w₂ were calculated according to thetechniques of Section II as 35 and 38, respectively. For records r₁ andr₂, the term M may be calculated by noting that the common tokens to theBusiness Name field for these records are “Lawn”, “Furniture” and“Corporation”, which have token weights of 10, 12 and 4, respectively.The sum of these weights, M, is equal to 10+12+4, or 26. For records r₁and r₂, the term N may be calculated by noting that the tokens in theBusiness Name field for these records that do not match are “Joe's” (inr₁) and “Abe's” (in r₂). These tokens have token weights 15 and 20,respectively. The sum of these weights, N, is equal to 15+20, or 25.Accordingly, applying these numbers to Equations 33-36 yields,respectively:

$\begin{matrix}\begin{matrix}{w_{1,2} = {{\max \left( {w_{1},w_{2}} \right)} \times \frac{2\; M}{{2\; M} + N}}} \\{= {{\max \left( {35,38} \right)} \times \frac{2 \times 26}{{2 \times 26} + 25}}} \\{= 25.7}\end{matrix} & {{Equation}\mspace{14mu} 37} \\\begin{matrix}{w_{1,2} = {{\min \left( {w_{1},w_{2}} \right)} \times \frac{{2M} - N}{{2\; M} + N}}} \\{= {{\min \left( {35,38} \right)} \times \frac{{2 \times 26} - 25}{{2 \times 26} + 25}}} \\{= 12.3}\end{matrix} & {{Equation}\mspace{14mu} 38} \\\begin{matrix}{w_{1,2} = {{\max \left( {w_{1},w_{2}} \right)} \times \frac{{2\; M} - N}{{2\; M} + N}}} \\{= {{\max \left( {35,38} \right)} \times \frac{{2 \times 26} - 25}{{2 \times 26} + 25}}} \\{= 13.3}\end{matrix} & {{Equation}\mspace{14mu} 39} \\{w_{1,2} = {{\min \left( {w_{1},w_{2},M} \right)} = {{\min \left( {35,38,\; 26} \right)} = 26}}} & {{Equation}\mspace{14mu} 40}\end{matrix}$

Next, calculations are presented according to Equations 33-36 withrespect to the pair of records r₁ and r₃ taken from Table X.1 and theshared multi token Business Name field. Note that, in this example forrecords r₁ and r₃, the terms w₁ and w₃ were calculated according to thetechniques of Section II as 35 and 27, respectively. For records r₁ andr₃, the term M may be calculated by noting that the common tokens to theBusiness Name field for these records are “Joe's”, “Furniture” and“Corporation”, which have token weights of 15, 12 and 4, respectively.The sum of these weights, M, is equal to 15+12+4, or 31. For records r₁and r₃, the term N may be calculated by noting that the only token inthe Business Name field for these records that does not match is “Lawn”(which appears in r₁ but not in r₂). This token has a token weight of10; accordingly, N is equal to 10 for records r₁ and r₃. Thus, applyingthese numbers to Equations 33-36 yields, respectively:

$\begin{matrix}\begin{matrix}{w_{1,3} = {{\max \left( {w_{1},w_{3}} \right)} \times \frac{2\; M}{{2\; M} + N}}} \\{= {{\max \left( {35,27} \right)} \times \frac{2 \times 31}{{2 \times 31} + 10}}} \\{= 30.1}\end{matrix} & {{Equation}\mspace{14mu} 41} \\\begin{matrix}{w_{1,3} = {{\min \left( {w_{1},w_{3}} \right)} \times \frac{{2\; M} - N}{{2\; M} + N}}} \\{= {{\min \left( {35,27} \right)} \times \frac{{2 \times 31} - 10}{{2 \times 31} + 10}}} \\{= 19.5}\end{matrix} & {{Equation}\mspace{14mu} 42} \\\begin{matrix}{w_{1,3} = {{\max \left( {w_{1},w_{3}} \right)} \times \frac{{2\; M} - N}{{2\; M} + N}}} \\{= {{\max \left( {35,27} \right)} \times \frac{{2 \times 31} - 10}{{2 \times 31} + 10}}} \\{= 25.3}\end{matrix} & {{Equation}\mspace{14mu} 43} \\{w_{1,3} = {{\min \left( {w_{1},w_{3},M} \right)} = {{\min \left( {35,27,31} \right)} = 27}}} & {{Equation}\mspace{14mu} 44}\end{matrix}$

Note that in this example, the blended weights associated with recordsr₁ and r₃ (provided in Equations 41-44) exceed the blended weightsassociated with records r₁ and r₂ (provided in Equations 37-40). Thus,based on the Business Name field contents alone, it is more likely thatrecord r₁ would be linked to record r₃ than to record r₂. Note furtherthat the Business Name field values of records r₁ and r₂ are closertogether than the Business Name field values of records r₁ and r₃ whencompared by naively using an edit distance measure. Thus, this exampleillustrates that the blended field weights may provide a better measureof match significance between multi token field values than a naïveapplication of edit distance, for example.

At block 1025, after the first iterations (for the token table and thedatabase at large), the exemplary technique may undergo a linkingprocess that generates a plurality of entity representations. The linkprocess may generally proceed as set forth in Sections I and II. Moreparticularly, Equations 3-6 may be used to compare pairs of records anddecide whether to link them. Each record may be compared to every otherrecord in the database, or to a set of records generated using blockingcriteria, such as the blocking criteria presented in Section VIII. Asdiscussed in Section I, comparing two records using, e.g., Equations3-5, may involve comparing individual field values to generate aprobability and then weighting such probability using a weight. Incomparing a multi token field as part of computing a match score, theprocess may proceed as follows. If the field values in the multi tokenfield are identical, then the associated probability may be set to one(1) and weighted using the field value weight for the entire multi tokenfield value, or, alternately, the field weight associated with the multitoken field. If, on the other hand, the field values in the multi tokenfield are not identical, then the comparison of the multi token fieldmay proceed as follows. The process may generate a probability that thefield values match according to techniques for generating suchprobabilities discussed herein and in the First Generation Patents AndApplications. As a particular non-limiting example, the probability maybe set to one (1), even though the field values are not exactlyidentical. The probability may be weighted by any of the blended weightsas disclosed in this section. The comparison may proceed to theremaining field values, a match score may be generated, the score may beused to determined whether to link the records as discussed in SectionI, and the records may be linked depending on the determination.

At block 1030, intermediate operations may be performed. Exemplary suchoperations (e.g., transitional linking, propagation, delinking) arediscussed in Section II.

Subsequent iterations (i.e., iterations subsequent to the firstiteration) may proceed in a manner similar to the first iteration. Inparticular, for each subsequent iteration, a token table may begenerated at block 1035 for each selected multi token field. Such atoken table may include DIDs or other indicia of linkage betweenrecords. The token table may be generated from scratch or by revisingthe token table from the prior iteration. That is, the token table fromthe prior iteration may be updated by altering the DIDs of records thatwere linked after the prior iteration. Continuing the example presentedin this section, and assuming for purposes of illustration that thelinking process that followed the first iteration linked records r₁ andr₃, (a portion of) the token table for a second iteration may appear aspresented below:

TABLE X.6 DID Token 1 Joe's 1 Lawn 1 Furniture 1 Corporation 2 Abe's 2Lawn 2 Furniture 2 Corporation 1 Joe's 1 Furniture 1 Corporation

At block 1040, and in general, iterations subsequent to the first mayproceed to compute token field value probabilities and token field valueweights for the records in the token table (e.g., Table X.6). That is,the subsequent iterations may proceed by calculating token probabilitiesand token weights. These may be computed using any of the techniquesdisclosed herein. Thus, subsequent iterations may count the number ofrecords in the token table that include a particular token, countingmultiple tokens that appear in the same original field as one, and thesecounts may be then divided by the total number of different DIDs in thetoken table, yielding token probabilities. Thus, at the end of eachsubsequent iteration, each token together with the field in which itoriginally appeared may be associated with a token probability, whichmay be calculated as the number of records with unique DIDs in the tokentable that include the token, divided by the total number of unique DIDsin the token table. In formal terms, the token probability may becalculated as, by way of non-limiting example:

$\begin{matrix}{{p_{f,t}(n)} = {\frac{c_{f,t}}{c}.}} & {{Equation}\mspace{14mu} 45}\end{matrix}$

In Equation 45, the term p_(f,t) (n) represents the n-th iteration tokenprobability associated with original field f and token t. The remainingterms in Equation 45 may be as described in reference to Equation 31,and may be determined at the time of the subsequent iteration.Accordingly, a given token probability produced by a subsequentiteration may be a probability that a record randomly chosen from the(original) database, after the prior iteration, contains the given tokenin the associated field. The field value probabilities may be convertedto field value weights according to, by way of non-limiting example:

w _(f,t)(n)=−log p _(f,t)(1).   Equation 46

Thus, at the end of each iteration, each token and the original field inwhich it appears may be associated with a token weight, each of whichmay be calculated from a corresponding token probability.

Each iteration may include separately subjecting the database at large(or a portion thereof) to an iteration according to, for example, thetechniques of Section II (e.g., using Equations 15-18). Such aniteration may generate revised field value probabilities and revisedfield value weights for the entirety of each multi token field value atblock 1045. Such parameters may be generated using the techniquesdescribed in Section II (e.g., Equations 15 and 16) and stored for lateruse according to any of the storage techniques disclosed herein. Eachsubsequent iteration may proceed to generate revised field probabilitiesand revised field weights associated with each multi token field. Suchparameters may be generated using the techniques described in Section II(e.g., Equations 17 and 18) and stored for later use according to any ofthe storage techniques disclosed herein. Thus, subjecting the databaseto each subsequent iteration may generate match weights for entire multitoken fields and multi token field values.

In each iteration, the token weights and multi token field value weightsmay be stored as discussed in this section above in reference to thefirst iteration, e.g., in tables or in the multi token fields of theoriginal records themselves.

At block 1050, each iteration may proceed to generate blended weights asdescribed in this section above in reference to the first iteration,e.g., according to Equations 33-36.

At block 1055, each iteration may be followed by a linking process. Sucha linking process may proceed as described in this section above.

At block 1060, intermediate operations may be performed. Exemplary suchoperations (e.g., transitional linking, propagation, delinking) arediscussed in Section II.

Block 1065 indicates that one or more of blocks 1035, 1040, 1045, 1050,1055 and 1060 may be iterated.

Thus, each iteration may generate more accurate blended weights, whichmay be used to link together records that correspond to the same entity.That is, each iteration may consolidate entity representations. It isexpected that at some point, the process stabilizes such that furtheriterations do not result in further linkages.

The iteration may halt after any number of iterations after any ofblocks 1035, 1040, 1045, 1050, 1055 or 1060. At block 1070, the blendedweights may be used to link records as discussed elsewhere herein.

Modifications and alterations to the process described in this sectionabove are discussed presently. Instead of, or in addition to,calculating token weights and multi token field value weights based onexact matches using e.g., the techniques of Section II, some embodimentsmay calculate token weights and multi token field value weights based onother techniques discussed herein. Such techniques include, e.g., thosedisclosed in Sections III and IV. Once token weights and multi tokenfield value weights are so calculated, such embodiments may use suchparameters to calculate blended weights according to, e.g., Equations33-36. Such token weights and multi token field weights may be storedin, by way of non-limiting example, separate tables, in one or morefields added to the original records or token tables, or embedded withinthe multi token fields themselves.

An example of using the techniques of Sections III and IV within thetechniques disclosed in this section is presented. (Note that Equations33-35 include terms N, which reflect a technique of Section IX.) Thisexample is presented in view of Tables X.1 and X.2. Further, thisexample is presented in the context of a first iteration; however, thisexample may be extended to subsequent iterations using the techniquesdisclosed herein. That is, this example is meant to illustrate one wayof incorporating the techniques of Sections III and IV into a firstiteration of an embodiment of a technique of the present section, butadditional configurations and further iterations fall within the scopeof the present disclosure. In order to utilize a technique according toSection IV, an additional proxy field may be added to the token table,resulting in a table such as presented below.

TABLE X.7 DID Token Proxy Token 1 Joe's J200 1 Lawn L500 1 FurnitureF653 1 Corporation C616 2 Abe's A120 2 Lawn L500 2 Furniture F653 2Corporation C616 3 Joe's J200 3 Furniture F653 3 Corporation C616

As illustrated in Table X.7, the proxy fields may be populated with aSOUNDEX code, or any other suitable code as discussed in Section IV.Additional columns may be added to accommodate one or more of: tokenweights produced according to the techniques of Section II (e.g., asdisclosed in relation to Table X.3, above), weights for the proxy tokens(e.g., as disclosed in Section IV), and weights for one or morereflexive symmetric distance measure and associated one or moredistances (e.g., as disclosed in Section III). An example of such atable appears below.

TABLE X.8 Proxy Token Proxy Token DID Token Weight Token Weightw_(f,v,D) ₁ _(,d) ₁ w_(f,D) ₂ _(,d) ₂ 1 Joe's 15 J200 12 4 7 1 Lawn 10L500 6 5 7 1 Furniture 12 F653 11 11 7 1 Corporation 4 C616 4 3 7 2Abe's 20 A120 16 18 7 2 Lawn 10 L500 6 5 7 2 Furniture 12 F653 11 11 7 2Corporation 4 C616 4 3 7 3 Joe's 15 J200 12 4 7 3 Furniture 12 F653 1111 7 3 Corporation 4 C616 4 3 7

In Table X.8, the Token Weight column contains the token weightscomputed in a first iteration according to a technique of Section II.Populating this column is discussed above in detail in this sectionabove. The Proxy Token field is, in this example, populated with SOUNDEXcodes for each corresponding token. As discussed in Section IV, othercodes instead of or in addition to SOUNDEX may be used. The Proxy Tokenweight column in Table X.8 is populated with weights for the proxy fieldvalues according to the techniques of Section IV. The w_(f,v,D) ₁ _(,d)₁ field contains field value weights for the tokens as computedaccording to the techniques of Section III. More particularly, theseweights may be computed using, for example, Equations 20 or 24. Theparameter f in this instance is the token field in the token table(column two in Table X.8), and the parameter v is the associated tokenitself. The term D₁ represents a selected reflexive and symmetricdistance function, which may be any such function consistent with thedisclosure of Section III. The w_(f,D) ₂ _(,d) ₂ field contains fieldweights for the tokens as computed according to the techniques ofSection III. More particularly, these weights may be computed using, forexample,

Equations 22 or 26. The parameters in this instance is the token fieldin the token table (column two in Table X.8), and the term D₂ representsa selected reflexive and symmetric distance function, which may be anysuch function consistent with the disclosure of Section III. In theexample presented above in reference to Tables X.7 and X.8, the weightsthat appear in Table X.8 may be computed as part of a first iteration ofan iterative process. Thus, each weight that appears in Table X.8 may becomputed as part of a single iteration, rather than requiring separateiterations. However, in some embodiments, separate iterations may beemployed.

Continuing the above example, the exemplary first iteration may beaccompanied by a first iteration of the entire original database (orportion thereof) in order to generate field value probabilities andfield value weights for each of the entire multi token field valuesaccording to the techniques of Section II are discussed above. Fieldprobabilities and field weights may also be generated. Processes forgenerating such parameters according to the techniques of Section II arediscussed above.

In addition, field value probabilities and field value weights may begenerated for each of the entire multi token field values according tothe same techniques that were applied to the token table (e.g., TableX.8). That is, a proxy token field may be added to the original recordsand populated with codes for the entirety of the selected Business Namefield values according to the same technique used to populate the ProxyToken field of Table X.8. Field value weights for these proxy fieldvalues maybe generated according to the techniques discussed in SectionIV, and these weights may be stored in a field added to the originalrecords or elsewhere, e.g., in a separate table or according to any ofthe storage techniques disclosed herein. Additionally, weights for theentire multi token field values may be computed according to the sametechniques used to compute the weights that appear in the last twocolumns of Table X.8, and the weights so generated may be stored infields added to the original records or elsewhere as discussed herein.Thus, at this point, each multi token field, in its entirety, each ofthe proxy field values for the selected multi token field, each of theconstituent tokens present in the multi token fields, and each tokenproxy field value may have multiple associated weights according to thetechniques used to populate the third, fifth, sixth and seventh columnsof Table X.8.

These parameters may be combined according to the techniques disusedabove in relation to Equations 33-36 in order to generate blendedweights. In some embodiments, only like weights are combined with likeweights in generating blended weights. For example, proxy token weightsmay be blended with weights for the entirety of the proxy field valuesappearing in the multi token fields according to the techniquesdiscussed above in relation to Equations 33-36. Likewise, token weightscomputed according to the formula for w_(f,v,D) ₁ _(,d) ₁ may be blendedwith weights for the entire multi token field values according to thesame technique (e.g., using Equations 20 or 24). Also token fieldweights computed according to the formula for w_(f,D) ₂ _(,d) ₂ may becombined with weights for the entire multi token field, or field values,according to the same technique (e.g., using Equations 22 or 26). Insome embodiments, one or more token weights may be combined with one ormore weight for the entire multi token field or field value in order togenerate blended weights, regardless as to the techniques used togenerate these parameters. These blended weights may be computed andstored or computed on the fly as part of a linking process.

Once the blended weights are generated, they may be used in a linkingprocess in a manner similar to that as discussed above in this section.The linking process may be followed by a transition linking process, apropagation operation, and a delinking operation. Exemplary suchprocedures are discussed above in Section II.

Additional iteration may follow. Each iteration may use the same ordifferent parameters and blended weights. Each iteration is expected tofurther consolidate entity representations until a stable point isreached.

Although DIDs are discussed above in an exemplary embodiment, alternatetechniques for linking records may be used with the appropriatemodifications to the techniques discussed in this section.

According to an embodiment of the invention, a method of generating arecord matching formula weight, where the record matching formula weightis specific to a particular field value associated with a particularfield, the particular field value comprising a plurality of tokens, ispresented. The method includes calculating, for each token comprisingthe particular field value, a probability that a record includes thetoken in the particular field, where a first plurality of probabilitiesare calculated. The method also includes calculating a first probabilityincluding the first plurality of probabilities. The method furtherincludes linking records in the database based at least in part on thefirst probability, where a plurality of entity representations aregenerated. The method further includes calculating, for each tokenincluding the particular field value, a probability that an entityrepresentation includes a record including the token in the particularfield, where a second plurality of probabilities are calculated. Themethod further includes calculating a second probability including thesecond plurality of probabilities. The method further includes linkingentity representations in the database based at least in part on thesecond probability. The method further includes retrieving informationfrom at least one record in the database.

Optional features of the above embodiment include the following. Themethod may further include iterating (1) the calculating, for each tokencomprising the particular field value, a probability, (2) thecalculating a second probability and (3) the linking entityrepresentations at least once prior to the retrieving. The method mayfurther include calculating a probability that two records match usingthe record matching formula, where the record matching formula includesa plurality of probabilities that two records match, where the weightsinclude the second probability.

According to an embodiment of the invention, a method of generating arecord matching formula weight, where the record matching formula weightis specific to a particular field and independent of any particularfield value in the particular field, where the particular field isconfigured to contain field values each comprising a plurality oftokens, is presented. The method includes calculating a plurality offirst probabilities, each of the plurality of first probabilitiesreflecting a likelihood that a record includes a particular token in theparticular field. The method also includes calculating a secondplurality of probabilities, each of the second plurality ofprobabilities including first probabilities associated with a fieldvalue associated with the particular field. The method further includescalculating a first weight including a weighted sum of the secondprobabilities. The method further includes linking records in thedatabase based at least in part on the first weight, where a pluralityof entity representations are generated. The method further includescalculating a third plurality of probabilities, each of the thirdplurality of probabilities reflecting a likelihood that an entityrepresentation includes a particular token in the particular field. Themethod further includes calculating a fourth plurality of probabilities,each of the fourth plurality of probabilities including thirdprobabilities associated with a field value associated with theparticular field. The method further includes calculating a secondweight including a weighted sum of the fourth probabilities. The methodfurther includes linking entity representations in the database based atleast in part on the second weight. The method further includesretrieving information from at least one record in the database.

XI. Exemplary Embodiments

A discussion of exemplary, non-limiting embodiments follows.

The embodiment begins by assembling a database or “master file.”(Throughout this disclosure, the terms “master file” and “database” aresynonymous.) Creating the database may include a process, such asprocess 200 of the First Generation Patents And Applications. Such aprocess typically initiates at a preparation phase, where incoming datamay be received from one or more data source and formatted to becompatible with the format of the master file, where the master filerepresents the database upon which queries may be performed. Theincoming data can include data from any of a variety of sources and haveany of a variety of heterogeneous formats. To illustrate, the incomingdata could include a data set from a motor vehicle registrationdatabase, where the information in the data set may be formatted andarranged in a proprietary way. Prior to inserting the motor vehicleregistration information into the master file, the information may needto be converted to a homogenous format consistent with the informationalready present in the master file. Accordingly, the preparation phaseincludes various processes to translate the incoming data into entityreferences for inclusion in the master file.

These processes may include, for example, deduplication (“dedup”) ofincoming data records, filtering of the incoming data to removeunrelated information, converting data fields from one format toanother, and the like. For example, the incoming data could include aname data field having a first name followed by a surname for eachrecord, whereas the master file could include separate first name andsurname data fields. The preparation phase, in this example, thereforemay include the step of separating the name data field of each record ofthe incoming data to a separate first name data field and surname datafield. After formatting the data of each record, the information in thedata fields of each record may be used to populate a correspondingproposed database record. Additional features of an exemplarypreparation phase are disclosed in the First Generation Patents AndApplications.

Once the database is set up, a linking phase may occur. The linkingphases may include features as disclosed in the First Generation PatentsAnd Applications, as disclosed in the present document, or a combinationof both.

A non-limiting, exemplary link phase is discussed presently. The processmay begin by implementing a technique according to Section VI. That is,for some or all records, field values that qualify as null valuesaccording to a technique of Section VI may be replaced with a singlefield value, for example, the empty field value. The process maycontinue by adding additional fields to the records as discussed inSections IV and V. The process may further continue by selecting a multitoken field from the records in the database and building a token tableas discussed in Section X.

The process may proceed to implement an iteration as disclosed inSection II. Note that the iteration of Section II may be viewed as aframework from within which other inventive techniques may beimplemented. Thus, an iterative process as discussed in Section II maycalculate match weights for one or more distance functions and one ormore distances according to a technique of Section III. Such weights maybe added to additional fields in each record. Each iteration maytherefore calculate a variety of field weights for a variety of fields.The process may further include, in the same or a separate iteration,calculating token weights for the token table as discussed in Section X.

Between each iteration, the database may undergo a linking operation.The linking operation may utilize one or more matching formulas asdiscussed in Section I in combination with a threshold set according toan administrator's determination of a suitable confidence level (e.g.,according to Equation 6 and the table in Section I). Such a linkingoperation may compare every record to every other record to which it isnot already linked. Alternately, the linking operation may compare everyrecord to every record generated according to a blocking criteria ofSection VIII to which it is not already linked. As discussed in SectionII and elsewhere, the matching formulas may utilize a variety ofweights. For example, the matching formulas may utilize negative matchweights according to a technique discussed in Section IX. It may utilizeany of the weights as generated according to techniques of Sections II-Vand X. It may utilize one or both of proxy fields and supplementalfields as disclosed in Sections IV and V respectively. Those additionalfields may be accounted for in the matching formula as discussed herein.Also between each iteration, the database may undergo several types ofprocessing. For example, between iterations, the database may undergo atransitional linking process, such as one or more of those discussed inthe First Generation Patents And Patent Applications or a techniquepresented in Section VII. Also between iterations, the database mayundergo a propagation operation, such as discussed in Sections II orVII. The database may further undergo a delinking operation such as oneor more that are disclosed in the First Generation Patents AndApplications.

After a suitable number of iterations, the database may be provided to auser for retrieval of information.

When additional information is added to the database, the processesdescribed herein may be iterated one or more additional times in orderto fully assimilate and link the additional information.

XII. Conclusion

Any of the techniques disclosed herein may be applied to a portion of adatabase as opposed to the entirety of a database.

The techniques discussed herein may be combined with any of thetechniques disclosed in the First Generation Patents And Applications.The inventors explicitly consider such combinations at the time offiling the present disclosure.

The equations, formulas and relations contained in this disclosure areillustrative and representative and are not meant to be limiting.Alternate equations may be used to represent the same phenomenadescribed by any given equation disclosed herein. In particular, theequations disclosed herein may be modified by adding error-correctionterms, higher-order terms, or otherwise accounting for inaccuracies,using different names for constants or variables, or using differentexpressions. Other modifications, substitutions, replacements, oralterations of the equations may be performed.

Certain embodiments of the inventions disclosed herein may output a morethoroughly linked database. Certain embodiments of the inventionsdisclosed herein may output any information contained in any record in adatabase.

Embodiments, or portions of embodiments, disclosed herein may be in theform of “processing machines,” such as general purpose computers, forexample. As used herein, the term “processing machine” is to beunderstood to include at least one processor that uses at least onememory. The at least one memory stores a set of instructions. Theinstructions may be either permanently or temporarily stored in thememory or memories of the processing machine. The processor executes theinstructions that are stored in the memory or memories in order toprocess data. The set of instructions may include various instructionsthat perform a particular task or tasks, such as those tasks describedherein. Such a set of instructions for performing a particular task maybe characterized as a program, software program, or simply software.

As noted above, the processing machine executes the instructions thatare stored in the memory or memories to process data. This processing ofdata may be in response to commands by a user or users of the processingmachine, in response to previous processing, in response to a request byanother processing machine and/or any other input, for example.

As noted above, the processing machine used to implement embodiments maybe a general purpose computer. However, the processing machine describedabove may also utilize any of a wide variety of other technologiesincluding a special purpose computer, a computer system including amicrocomputer, mini-computer or mainframe for example, a programmedmicroprocessor, a micro-controller, a peripheral integrated circuitelement, a CSIC (Customer Specific Integrated Circuit) or ASIC(Application Specific Integrated Circuit) or other integrated circuit, alogic circuit, a digital signal processor, a programmable logic devicesuch as a FPGA, PLD, PLA or PAL, or any other device or arrangement ofdevices that is capable of implementing the steps of the processes ofthe invention. In particular, the hardware described in the FirstGeneration Patents And Applications may be used for any embodimentdisclosed herein. A cluster of personal computers or blades connectedvia a backplane (network switch) may be used to implement someembodiments.

The processing machine used to implement the invention may utilize asuitable operating system. Thus, embodiments of the invention mayinclude a processing machine running the Microsoft Windows™ Vista™operating system, the Microsoft Windows™ XP™ operating system, theMicrosoft Windows™ NT™ operating system, the Windows™ 2000 operatingsystem, the Unix operating system, the Linux operating system, the Xenixoperating system, the IBM AIX™ operating system, the Hewlett-Packard UX™operating system, the Novell Netware™ operating system, the SunMicrosystems Solaris™ operating system, the OS/2™ operating system, theBeOS™ operating system, the Macintosh operating system, the Apacheoperating system, an OpenStep™ operating system or another operatingsystem or platform.

It is appreciated that in order to practice the method of the inventionas described above, it is not necessary that the processors and/or thememories of the processing machine be physically located in the samegeographical place. That is, each of the processors and the memoriesused by the processing machine may be located in geographically distinctlocations and connected so as to communicate in any suitable manner.Additionally, it is appreciated that each of the processor and/or thememory may be composed of different physical pieces of equipment.Accordingly, it is not necessary that the processor be one single pieceof equipment in one location and that the memory be another single pieceof equipment in another location. That is, it is contemplated, forexample, that the processor may be two ore more pieces of equipment intwo different physical locations. The two ore more distinct pieces ofequipment may be connected in any suitable manner. Additionally, thememory may include two or more portions of memory in two or morephysical locations.

To explain further, processing as described above is performed byvarious components and various memories. However, it is appreciated thatthe processing performed by two or more distinct components as describedabove may, in accordance with a further embodiment of the invention, beperformed by a single component. Further, the processing performed byone distinct component as described above may be performed by two ormore distinct components. In a similar manner, the memory storageperformed by two or more distinct memory portions as described abovemay, in accordance with a further embodiment of the invention, beperformed by a single memory portion. Further, the memory storageperformed by one distinct memory portion as described above may beperformed by two or more memory portions.

Further, various technologies may be used to provide communicationbetween the various processors and/or memories, as well as to allow theprocessors and/or the memories of the invention to communicate with anyother entity; e.g., so as to obtain further instructions or to accessand use remote memory stores, for example. Such technologies used toprovide such communication might include a network, the Internet,Intranet, Extranet, LAN, an Ethernet, or any client server system thatprovides communication, for example. Such communications technologiesmay use any suitable protocol such as TCP/IP, UDP, or OSI, for example.

As described above, a set of instructions is used in the processing ofembodiments. The set of instructions may be in the form of a program orsoftware. The software may be in the form of system software orapplication software, for example. The software might also be in theform of a collection of separate programs, a program module within alarger program, or a portion of a program module, for example. Thesoftware used might also include modular programming in the form ofobject oriented programming. The software tells the processing machinewhat to do with the data being processed.

Further, it is appreciated that the instructions or set of instructionsused in the implementation and operation of the invention may be in asuitable form such that the processing machine may read theinstructions. For example, the instructions that form a program may bein the form of a suitable programming language, which is converted tomachine language or object code to allow the processor or processors toread the instructions. That is, written lines of programming code orsource code, in a particular programming language, are converted tomachine language using a compiler, assembler or interpreter. The machinelanguage is binary coded machine instructions that are specific to aparticular type of processing machine, e.g., to a particular type ofcomputer. The computer understands the machine language.

Any suitable programming language may be used in accordance with thevarious embodiments of the invention. Illustratively, the programminglanguage used may include Enterprise Control Language (“ECL,” availablefrom LexisNexis), assembly language, Ada, APL, C, C++, dBase, Fortran,Java, Modula-2, Pascal, REXX, Visual Basic, and/or JavaScript, forexample. Further, it is not necessary that a single type of instructionsor single programming language be utilized in conjunction with theoperation of the system and method of the invention. Rather, any numberof different programming languages may be utilized as is necessary ordesirable.

Also, the instructions and/or data used in the practice of the inventionmay utilize any compression or encryption technique or algorithm, as maybe desired. An encryption module might be used to encrypt data. Further,files or other data may be decrypted using a suitable decryption module,for example.

It is to be appreciated that the set of instructions, e.g., thesoftware, that enables the computer operating system to perform theoperations described above may be contained on any of a wide variety ofmedia or medium, as desired. Further, the data that is processed by theset of instructions might also be contained on any of a wide variety ofmedia or medium. That is, the particular medium, i.e., the memory in theprocessing machine, utilized to hold the set of instructions and/or thedata used in the invention may take on any of a variety of physicalforms or transmissions, for example. Illustratively, the medium may bein the form of paper, paper transparencies, a compact disk, a DVD, anintegrated circuit, a hard disk, a floppy disk, an optical disk, amagnetic tape, a RAM, a ROM, a PROM, a EPROM, a wire, a cable, a fiber,communications channel, a satellite transmissions or other remotetransmission, as well as any other medium or source of data that may beread by the processors of the invention.

Further, the memory or memories used in the processing machine thatimplements an embodiment may be in any of a wide variety of forms toallow the memory to hold instructions, data, or other information, as isdesired. Thus, the memory might be in the form of a database to holddata. The database might use any desired arrangement of files such as aflat file arrangement or a relational database arrangement, for example.

In some embodiments, a variety of “user interfaces” may be utilized toallow a user to interface with the processing machine or machines thatare used to implement the embodiment. As used herein, a user interfaceincludes any hardware, software, or combination of hardware and softwareused by the processing machine that allows a user to interact with theprocessing machine. A user interface may be in the form of a dialoguescreen for example. A user interface may also include any of a mouse,touch screen, keyboard, voice reader, voice recognizer, dialogue screen,menu box, list, checkbox, toggle switch, a pushbutton or any otherdevice that allows a user to receive information regarding the operationof the processing machine as it processes a set of instructions and/orprovide the processing machine with information. Accordingly, the userinterface is any device that provides communication between a user and aprocessing machine. The information provided by the user to theprocessing machine through the user interface may be in the form of acommand, a selection of data, or some other input, for example.

As discussed above, a user interface is utilized by the processingmachine that performs a set of instructions such that the processingmachine processes data for a user. The user interface is typically usedby the processing machine for interacting with a user either to conveyinformation or receive information from the user. However, it should beappreciated that in accordance with some embodiments of the system andmethod of the invention, it is not necessary that a human user actuallyinteract with a user interface used by the processing machine of theinvention. Rather, it is also contemplated that the user interface ofthe invention might interact, e.g., convey and receive information, withanother processing machine, rather than a human user. Accordingly, theother processing machine might be characterized as a user. Further, itis contemplated that a user interface utilized in the system and methodof the invention may interact partially with another processing machineor processing machines, while also interacting partially with a humanuser.

It will be readily understood by those persons skilled in the art thatembodiments of the present inventions are susceptible to broad utilityand application. Many embodiments and adaptations of the presentinventions other than those herein described, as well as manyvariations, modifications and equivalent arrangements, will be apparentfrom or reasonably suggested by the present invention and foregoingdescription thereof, without departing from the substance or scope ofthe invention.

Accordingly, it is to be understood that this disclosure is onlyillustrative and exemplary and is made to provide an enablingdisclosure. Accordingly, the foregoing disclosure is not intended to beconstrued or to limit the present invention or otherwise to exclude anyother such embodiments, adaptations, variations, modifications orequivalent arrangements.

1. A computer implemented method of detecting and treating one or morefield values as null field values in records of an electronic database,the computer implemented method comprising: selecting a field of therecords; determining, using a programmed computer, a frequency for eachfield value of the field, the frequency comprising an amount of therecords in which the field value of the field appears; calculating,using a programmed computer, a critical frequency for the field based onthe frequency of each field value of the field; and treating, using aprogrammed computer, each field value of the field with a frequency thatis greater than the critical frequency of the field as a null fieldvalue and each field value of the field with a frequency that is lessthan the critical frequency of the field as a non-null field value. 2.The computer implemented method of claim 1, wherein the calculatingcomprises ordering the frequencies of the field values of the fieldaccording to frequency along a horizontal axis of a histogram of the oneor more field values of the field.
 3. The computer implemented method ofclaim 2, wherein the calculating further comprises calculating adifference between adjacent frequencies in the histogram.
 4. Thecomputer implemented method of claim 1, wherein the calculatingcomprises calculating f(x)=g(x)−g(x+1), wherein f is a function of thefrequency differences based on a function g that is defined by thefrequency of each field value of the field.
 5. The computer implementedmethod of claim 4, wherein the critical frequency of the field isdetermined by determining the first point at which a derivative of the ffunction changes from a first sign to a second sign.
 6. The computerimplemented method of claim 4, wherein the critical frequency of thefield is determined by determining the first point at which a derivativeof the f function changes from a positive sign to a negative sign. 7.The computer implemented method of claim 4, wherein the criticalfrequency of the field is determined by determining the first point atwhich a derivative of the f function changes from a negative sign to apositive sign.
 8. The computer implemented method of claim 4, whereinthe critical frequency of the field is determined by determining thefirst point at which a derivative of the f function equals zero.
 9. Thecomputer implemented method of claim 4, wherein the critical frequencyof the field is determined by determining the first point at which aderivative of the f function decreases below a threshold.
 10. Thecomputer implemented method of claim 1, wherein the treating furthercomprises treating each field value of the field with a frequency thatis equal to the critical frequency of the field as a null field value.11. The computer implemented method of claim 1, further comprisingreplacing one or more field values treated as null field values with acanonical null field value.
 12. A computer implemented system ofdetecting and treating one or more field values as null field values inrecords of an electronic database, the computer implemented systemcomprising: a first computing apparatus configured to select a field ofthe records; a second computing apparatus configured to determine afrequency for each field value of the field, the frequency comprising anamount of the records in which the field value of the field appears; athird computing apparatus configured to calculate a critical frequencyfor the field based on the frequency of each field value of the field;and a fourth computing apparatus configured to treat each field value ofthe field with a frequency that is greater than the critical frequencyof the field as a null field value and each field value of the fieldwith a frequency that is less than the critical frequency of the fieldas a non-null field value.
 13. The computer implemented system of claim12, wherein the third computing apparatus is configured to calculate byordering the frequencies of the field values of the field according tofrequency along a horizontal axis of a histogram of the one or morefield values of the field.
 14. The computer implemented system of claim13, wherein the third computing apparatus is further configured tocalculate by calculating a difference between adjacent frequencies inthe histogram.
 15. The computer implemented system of claim 12, whereinthe third computing apparatus is configured to calculate by calculatingf(x)=g(x)−g(x+1), wherein f is a function of the frequency differencesbased on a function g that is defined by the frequency of each fieldvalue of the field.
 16. The computer implemented system of claim 15,wherein the critical frequency of the field is determined by determiningthe first point at which a derivative of the f function changes from afirst sign to a second sign.
 17. The computer implemented system ofclaim 15, wherein the critical frequency of the field is determined bydetermining the first point at which a derivative of the f functionchanges from a positive sign to a negative sign.
 18. The computerimplemented system of claim 15, wherein the critical frequency of thefield is determined by determining the first point at which a derivativeof the f function changes from a negative sign to a positive sign. 19.The computer implemented system of claim 15, wherein the criticalfrequency of the field is determined by determining the first point atwhich a derivative of the f function equals zero.
 20. The computerimplemented system of claim 15, wherein the critical frequency of thefield is determined by determining the first point at which a derivativeof the f function decreases below a threshold.
 21. The computerimplemented system of claim 12, wherein the fourth computing apparatusis further configured to treat each field value of the field with afrequency that is equal to the critical frequency of the field as a nullfield value.
 22. The computer implemented system of claim 12, furthercomprising a fifth computing apparatus configured to replace one or morefield values treated as null field values with a canonical null fieldvalue.